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ABSTRACT 


This thesis evaluated the first iteration of Navy Postgraduate School (NPS) on¬ 
line course ‘Space Systems - Technology and Applications’. The thesis goal was to 
evaluate: course design; course effectiveness; student expectations and motivation before 
and affective reactions after the course; and typical patterns of on-line learning behavior 
according to Long’s theory. Ultimately, the thesis intended to derive lessons learned and 
recommendations for future distributed learning (DL) courses and future research in this 
area. 

There were thirty regular, resident NPS students in the sample. Data was collected 
from nine sources: four on-line questionnaires. Student Opinion Forms, on-line tracking, 
final grades, interviews with students, and interviews with the instructor. 

Even though the course first coiurse taught by this instructor, it was successful. 
Student expectations, motivation, affective and utility reactions were predominantly 
positive. For students, the most important advantage of the course was time flexibility 
and convenience. Students and the instructor agreed about central role of 
interactivity/feedback in a DL, which should be included into models of training 
effectiveness evaluation. Long-Dziuban’s protocol results were surprising, but solid 
conclusions cannot be made widiout focused research. Results provided several 
reco mme ndations about the course design, pedagogical improvements, instructional 
counseling, and future research. 
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I. INTRODUCTION 


This chapter provides the reader with the backgroxmd that led to the creation of 
this thesis, outlines the theoretical models that guided and directed the research, supplies 
the thesis purpose and the research questions, briefly reviews the thesis scope and 
methodology, and provides an overview on the organization of the study. 

This thesis tests a model of a web-based course evaluation, based on qjecific 
theoretical bases, which are described later in this chapter and in Chapter H. This model 
determines the research questions of this thesis, its methodology, and the study’s 
organization. 

My intention is to evaluate the utility of the model through the qualitative and 
quantitative analysis of data collected in this case study research in the context of 
relationships and factors that this model would predict. Lessons learned from the case 
study should primarily have practical applicability. However, the study might also 
provide conclusions in terms of knovm models and their possible modifrcations and 
improvements. 

A. BACKGROUND 

This section provides a brief description of events, circumstances, and theoretical 
concepts that explain the need for this research. 

Several years ago, the Chief of Naval Education and Training (CNET), U.S. 
Navy, intending to enhance education and training opportunities for Navy personnel and 
being aware of an increasing importance of distributed learning (DL) in contemporary 
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education and training, initiated an idea for a pilot project. The project did not have an 
official name, but in fiscal year 1999 all stakeholders agreed to the project and CNET 
ensured its funding. Decision-makers concluded that the Naval Postgraduate School 
(NPS) was an excellent place to run the pilot program. NFS agreed to transform several 
of its courses in the Information Systems and Operations (ISO) curriculum^ into web- 
based, on-line courses and to delivere them. In the first phase, the program would include 
two courses, one of which was ‘Space Systems - Technology and Applications’ taught in 
the Space Systems Academic Group. In total, the plan was to gradually transform twelve 
to fourteen courses to DL versions. By the end of 2000 foxn courses were on-line, and six 
more were planned for 2001. 

The reasons for such broad distributed learning initiati ves in the military are 
understandable. The modem military organization is very information-oriented and 
highly training-demanding because of the rapid development of military technologies. 
The vision for future battlefields is interactive and network-centered with 
communications playing a crucial role. Distributed learning is one of the inevitable, 
powerful training tools the military needs to obtain such a vision. During the last 10 
years, U.S. and foreign universities, other educational iostitutions, and companies in 
general have instituted DL courses and programs. Progress in this instmctional area has 
been very rapid. 

Before going further, I have to precisely define the central theme in this thesis: 
distributed learning. Bates (1999), in his excellent book ‘ Manag in g Technological 


^ ISO includes Information Operations and Infonnadon Warfare curricula. 
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Change’ (p. 27), quotes the definition of distributed learning by the Institute for 
Academic Technology, University of North Carolina: 

A distributed learning environment is a learner-centered approach to 
education, which integrates a number of technologies to enable 
opportunities for activities and interaction in both asynchronous and real¬ 
time modes. The model is based on blending a choice of appropriate 
technologies with aspects of campus-based delivery, open le arning 
systems, and distance education. The approach gives instructors the 
flexibility to customize learning enviromnents to meet the needs of diverse 
students populations, while providing both high-quality and cost-effective 
learning. 

Later in his text. Bates (1999) continues elaborating on this terminology (p. 28): 

New technologies are leading to major structural c hang es in the 
management and organization of teaching. These developments are 
increasingly being referred to in the United States and Canada as 
distributed learning, in the United Kingdom as networked learning, and in 
Australia as flexible learning. 

Belanger and Jordan (2000, pp. 49-53, 71-86) divide computer-aided instruction 
(CAI) into four main groups: 

• Web-based training (WBT) 

• Teleconferencing 

• Videotape instructions 

• Video tele tr aining (VTT) 

This thesis is focused on WBT. In Belanger and Jordan WBT, with the Internet as 
a primary medium for delivery of instruction, is then divided into: 

• Web Enhanced Courses - traditional classroom combined with Web 
. technologies 

• Web Managed Courses — where all course managem ent, testing, 
collaboration, instructional material, etc. are on-line 

• Web Delivered Courses - all materials available through the Web, 
including tools for real-time collaboration and interactivity 

• Hybrid Delivery - partial Web and partial CD-ROM course delivery 
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This conceptual division of WBT still does not provide a sufficiently precise 
definition. Williams, Paprock, and Covington (1999) provide a developmental continuum 
of distance learning, where the third level of development is closest to the term 
‘distributed learning’ (Figure 1). 


Level 3 (1990s to 21^ 

Century) 

Highly interactive 

^ Digital TV/Multitasking systems 
Software/TV network collaboration 

1 Multimedia CBT/CD-ROM/laser disk 

Hybrid networks 

t Intemet/WEB/on-line video 

Virtual classrooms 

Business, home, travel, continual 


/ 

Level 2 (1960s to 1990s) / 

Passive to moderately active ^ 

Level 1 (1880s to ^ 

1960s) Passive ^ ^ 

Two-way mteractive audio/video 

PC teletraining via the bulletin board system 
Electronic mail, computer-mediated 
conferences (CMCs) 

Computer-based training (CBT) 

Satellite video teletraining (one-way 


video/audio) 

^ ^ ^ AudioWideotape 

Audio teletraining (two-way audio) 

Radio 

Correspondence 

Telematics (audiographics) 


Figure 1. Distance Education Continuum [From: Williams, Paprock, and Covington 

(1999)]. 

This continuum shows the way multi-media based learning has evolved through 

the years, highlighting the development of interactive delivery enviromnents. Level 1, the 

most passive form of distance learning, consists of printed material, audio/videotapes, 

and radio transmission. This type of learning is asynchronous because the instructor and 

student tr ansmi t messages only one-way and receive responses, after a lengthy delay. 

Level 2 is passive to moderately active, and includes two-way audio training, one-way 

video/two-way audio tele-training, computer-based training (CBT), CD-ROMs, laser 

disk, e-mail, computer-mediated conferences, audio graphics, laser disks, and other 

similar media. It is considered synchronous. Level 3 is a hi^y interactive learning 
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environment, and the content of a course determines which technologies would be used 
as a primary instructional form of delivery. At this level, hybrid environments and all 
distributed learning technologies mentioned above, includin g the Internet and World 
Wide Web, are combined into a ‘virtual classroom’. 

Moskal and Dziuban (2000, p. 2) quote Moore and Kearsley’s definition of 
distance education: 

[Distance education is] planned learning that normally occurs in a 
different place fix)m teaching and as a result requires special techniques of 
course design, special instractional techniques, special methods of 
communication by electronic or other technology, as well as special 
organizational and administrative arrangements. 

In the same text (p. 3), Moskal and Dziuban added an important note: ‘University 
of Central Florida (UCF) uses the term distributed learning because courses offered 
involve distributing instruction using various media to students that may not, by 
definition, be at a distance’. In other words, DL, as defined by UCF, is mostly web-based 
and on-line, but may include videotape, two-way interaction television, and other similar 
media. 


Hartman, Dziuban, and Moskal (1999, p. 2) further explain how the University of 
Central Florida defines and divides asynchronous learnin g networks (ALN), which is 
very close to the term ‘distributed learning’ used in this thesis: 

ALN activity at UCF is characterized as one of three types. Web-enhanced 
(‘E’) courses are those using the Web in a useful but incidental manner. 
Typically, on-line materials are provided as supplemental or enrichment 
resources. All ‘E’ courses use WebCT as a repository for course materials, 
resource links, or for conferencing and e-mail. Media-enhanced (‘M’) 
courses make significant xise of the Web as an integral element of course 
delivery. Currently, some ‘M’ courses have a reduced number of class 
meetings (reduced seat time), using ALN activities to replace traditional 
class meetings. Fully-based courses are designated ‘W’, and have no 
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required class meetings. Some have an optional pre-course orientation 

and/or required final examination. 

This definition is much more appropriate for the needs of this thesis, because the 
course this thesis examines was designed and delivered based on the “UCF ‘W’ course 
model”. From here on, by DL (unless stated differently) I mean asynchronous, web- 
based course/training, but with the small possibility of some non - on-line content. The 
course this thesis assesses is closest to the ‘W’ model stated above. By synchronicity I 
mean that time must be scheduled for the on-line meetings to be held, with all 
participants available at that time. 

B. ON-LINE EDUCATION 

Students of all ages and instructors worldwide are witnessing an explosion of on¬ 
line education - evidence of this statement is visible in mass media, campuses, and in 
everyday life. The best example is the British Open University (http ://www.open.ac.ukl 
the world’s largest and most successful DL institution. 

The following examples are listed in Kearsley’s ‘Online Education’ (2000). One 
of the U.S. leaders working with schools and creating worthwhile learning programs is 
NASA (http://www.nasa.gov) . which provides exciting help in its subject matter to 
school children and adults. Companies heavily involved in school networking are, for 
instance. Pacific Bell (http://www.kn.pacbell.com) and Hewlett-Packard 
(http://www.hp.com) . 

It is impossible to list all of the other good examples of specific areas in on-line 
learning. There are sites for the senior learning community (http://www.seniomet.org) . 
for the disabled (http://vyww/rit.edu/~easi) . for teachers (http://www.webteacher.org) . and 
even about how to learn on-line (http://www.mtsu.edu/~studskl or 
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http://wvvw.fami1veducation.comy Also, there are paper and cyber periodicals and 
magazines devoted to on-line education, like ‘Syllabus’ (see httD://www.svllabus.comy 
and numerous conferences every year. 

The web-based version of SS3011 ‘Space Systems - Technology and 
Applications’ served as a pilot-course and a beta test to evaluate if course material could 
be delivered effectively via the Internet. Accordingly, the specific intention of the thesis 
is to examine the course, to evaluate its outcomes, to analyze how effective it was in 
terms of meeting course objectives, and to identify the problems, file obstacles, and other 
major factors, which played si gnifi cant roles in the effectiveness of the course. 

The course was the first designed and taught by the instructor. Commander Susan 
Higgins. It was the first on-line web-based course prepared and delivered in the Space 
Systems Academic Group at NFS. Prior to teaching file course, the instructor participated 
in a faculty training course entitled ‘Interactive Distributed Learning for Technology- 
Mediated Course Delivery’ (IDL 6543), at the University of Central Florida, Orlando, 
FL. 

UCF began training feculty to design and develop on-line courses in Time 1996, 

and over 150 UCF faculty have attended the IDL 6543 course. The Course Development 

and Web Services Department at UCF not only helps faculty in integrating technology 

and media to transform the learning process but also provides multimedia courseware, 

software and databases, learner support for success in technology-mediated classes, 

support web site development including the University’s primary site, and research and 

develop advanced technological applications (Barbara Truman-Davis: Brochure on 

Course Development & Web Services, UCF [2000]). The DDL course’s site is located at 
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http://reach.ucf.edxi/~idl6543. learner support at http://reach.ucf.edu/~coursdev/1 earning 
and http://reach.ucf.edu/~coursedev/cdroni. and information about WebCT, UCF web- 
based coinse management tool is at http://reach.ucf.edu/~coursedev/webct/sunnort.html 
(UCF [2000]). More information about WebCT can be foxmd at http://www.webct.com 
(UCF [1999]). 

The ‘UCF model’ of distributed learning is well known and recognized nationally 
and internationally. UCF received an APOQ-SHEEO (American Productivity and Quality 
Center and the State Higher Education Executive Officers) faculty development award in 
1998. The United States Distance Learning Association (USDLA) awarded UCF in 1999 
the ‘Excellence in Distance Learning Programming Award in Higher Education for its 
educational use of distance learning media, including die World Wide Web, interactive 
television, radio, and TV tapes. Also, in 1999 it was granted $200,000 from the Pew 
Learning and Technology Program at the Center for Academic Transformation as part of 
the Pew Grand Program in Coxirse Redesign. 

C. PURPOSE 

This thesis evaluates the SS3011 course ‘Space Systems - Technology and 
Applications’ taught during the summer quarter of academic year 2000 at NPS. My goal 
was to employ a case study approach that embodied components of a theoretical model of 
web-based learning effectiveness. 

The research evalxiates the following: 

• Course design - course objectives, content, and coxirse activities as well as 
pedagogical strategies xised in that design. 

• Course effectiveness in terms of student learning. 

• Student expectations and motivation before the course and affective 
reactions after taking the coxirse 
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• Typical patterns of online learning behavior. 

• Student confidence in applying skills and competencies in fiiture jobs 

Most importantly, this research integrates the above-mentioned variables into an 

interactive system so as to imderstand better the influences of these factors on each part 
of the learning process in the course. 

D. RESEARCH QUESTIONS 

Primary research questions include the following: 

• How successful was the web-based SS3011 course design in meeting 
learning outcomes? 

• What were the students’ expectations, motivation, on-line behavior 
patterns, cognitive outcomes, afifective-type reactions, and utility-type 
reactions during and after the course? 

• What recommendations for course improvement, and what lessons learned 
(in general) coiild be derived fi'om the case analysis of this course? 

Secondary research questions: 

• How effective were the course organi 2 ation, pedagogy, and technology in 
meeting course objectives? 

• What are recommendations for future evaluations and research on this 
course and other DL courses in future? 


E. METHODOLOGY REMARKS 

The process of collecting data necessary for this thesis was shaped by the 
following factors: 

• The ongoing course was the primary priority. The research required 
limited interference with the learning process, i.e. not disturbing either the 
instructor or the students. 

• Students voluntarily cooperated in the research. If any student, for 
whatever reason, refused to participate at any time during &e course, that 
decision was honored. None of the students refused participation, but 
some skipped some of the questionnaires and the interview. This 
phenomenon of students skipping or not giving feedback is an interesting 
issue itself and will be addressed in other chapters. 
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• I was obligated to guarantee the protection of the students’ identity and 
proper use of all collected data. In accordance with this element of the 
research, I do not mention student names in the thesis. 

• The number of students was not large - thirty-one students successfully 
finished the course. Consequently, this thesis uses the case study approach 
with the intention of doing qualitative and quantitative analysis. 

• The sample had to be limited only to students who met following 
important characteristics: they were regular students of the course and 
later had successfully completed it, and they were resident students at 
NFS. 


There were four main sources of data relevant for this case study: 

• Demographic data, student biographies, fiieir grades, all student activity 
tracking data collected in software used in the course (WebCT), and 
Student Opinion Forms (SOFs). 

• Evaluation questionnaires put on-line only for the piupose of this thesis. 
There were four such instruments (Appencfices A throu^ D). 

• Individual semi-structured interviews I conducted with 17 students after 
the course was over (Appendix E). 

• Extensive interviews I conducted with the instructor, mostly after the 
course 

Detailed illustrations and discussions on all these soxirces and data are in Chapter 
in (Methodology) and in Chapter IV (Results and Data Analysis). 

F. ORGAISIZATION OF THE THESIS 

I review the relevant literature in Chapter n. Chapter HI deals with the 

metiiodology and data collecting issues. Chapter IV describes and analyzes all coUected 
data. It is followed by a discussion and analysis of the results and their implications in 
Chapter V, and by conclusions and recommendations in Chapter VT. 
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n. LITERATURE REVIEW 


A HISTORICAL PREDECESSORS AND THE INTERNET 

The U.S. President Dwight Eisenhower established the Advanced Research 

Project Agency (ARPA) in 1958 as part of the Defense Dq)artment in order to improve 
the new national commitment to science and to bolster national defense (Moschovitis and 
others, 1999, pp. 35-36). ARPA began to explore the possibilities of information 
processing and computers under the visionary leadership of Joseph C. D. Licklider. 
Searching for new ways to improve human productivity, ARPA conducted extensive 
research on the feasibility of connecting computers across long distances, hi 1965, Larry 
Roberts connected his computer in the Boston area via a phone line with another in 
California, which was the entry into an entirely new era: the era of the Internet. 

Licklider and his team imagined a universal computer network. Ted Nelson 
coined the terms ‘hypertext’ and ‘hyperlink’ in 1965, and in 1969 ARPAnet is bom: the 
first multiple-site computer network, sponsored by the U.S. Department of Defense. 

ARPAnet started as an experimental, four-computer network. By 1971, ARPAnet 

linked almost two dozens sites, including MIT and Harvard. By 1974, fiiere were over 

200 sites. During the 1980s, more and more computers using different operating systems 

were connected. In 1983, the military portion of ARPAnet was moved onto MELnet, and 

ARPAnet was officially disbanded in 1990. In the late 1980s, the National Science 

Foundation’s NSFnet began its own network and allowed everyone to access it. It was, 

however, primarily the domain of ‘techies’, computer-science graduates, and university 

professors. Finally, in 1991 Tim Berners-Lee developed the World Wide Web at CERN 

(Centre Emopean pour la Recherche Nucleaire — European Laboratory for Particle 
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Physics) in Geneva. This event definitely opened the door to the unlimi ted development 
of the Internet society. 

According to the newest surveys (Nielsen/NetRatings, o nlin e resource, December 
2000), today over 153 million Americans use the Internet from their homes. Accesses to 
the Internet increases by 5.3% monthly, and trends are similar in many other countries 
such as the United Kingdom, Japan, and Scandinavian countries. The process of gradual 
integration of telephony, radio broadcasting, and TV with the Internet is rapidly 
increasing. It is predicted that by 2005 over one billion people may be connected to the 
Internet (Simon & Schuster New Millennium Encyclopedia and Home Reference 
Library, 2000, topic ‘What is the Internet’). 

The speed at which the Internet is spreading is incredible. Radio existed 38 years 
before 50 million people tuned in. It took television 13 years to reach 50 million viewers. 
After the introduction of the first personal computer kit, it took 16 years before 50 million 
people were using one. However, within only four years of the Internet being available to 
the general public, 50 million people were connected (Simon & Schuster, same topic). 

B. EVALUATION OF DL EFFECTIVENESS 

Many practitioners have recognized evaluation as an integral part of a lifecycle 

methodology in any instructional systems design (ISD) model. Course evaluation is a 

consistent part of the ‘UCF model’ of DL practice also. 

Belanger and Jordan (2000) describe the following model, called ‘ADDIE’, as 
most commonly used. It includes five key phases: 
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Develop instructional objectives and 
strategies. Establish instructional content 
and create storyboards. 

Identify requirements, resources, issues, 
and constrains. 

Program interfaces, databases, 
communication facilities for text, audio, 
video. Test and document. Develop support 
procedures. 

Delivering of the course. 


Evaluate effectiveness of course and 
delivery mechanisms. 


Figure 2. ADDIE Model for DL Projects [From: Belanger and Jordan (2000), p. 90]. 

Even a quick examination of relevant literature shows that there is a large amount 
of research on the effectiveness of web-based or web-enhanced education and training. 
However, few researchers have tried to capture several aspects of the process 
simultaneously. An education or tr ainin g process is a complex, multidimensional activity. 
Concentrating on only some aspects of the process may miss important, subtle factors 
important for overall effectiveness. 

There are not many theoretical models analyzing training effectiveness. The most 
useful are Kirlqpatrick’s Four Levels of Evaluation model, Marshall and Shriver’s Five 
Levels of Evaluation Model, and Van Slyke’s Framework. This thesis anchors these 
tr aining effectiveness models, using SS3011 as a data source, to determine which of these 
models is effective in assessing web-based instruction. 

Kraiger, Ford and Salas (1993) offered the following classification of leammg 
outcomes: 
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LEARNING 



. Figure 3. Kraiger, Ford and Salas’ Classification of Learning Outcomes. [From: 

Kraiger, Ford & Salas (1993), pp. 311-328]. 

Their model systematizes learning outcomes into three basic clusters and analyzes 
each of them in depth. Cognitive outcomes in their model are very close to ICirkpatrick’s 
knowledge transfer, affective outcomes close to learner reactions, and skill-based 
outcomes to behavior transfer. 


Kirkpatrick’s model [AUiger and others (1997), pp. 342-343; Belanger and others 
(2000), pp. 191-192] is still the most influential and is a starting point for other models. It 
differentiates between four categories of training criteria: Learner reactions; learning or 
knowledge transfer; behavior transfer; and organizational impact. Researchers often use 
this model to evaluate the impact of the learning experience at affective, cognitive, 
behavioral, and organizational levels. 


14 






According to Kirkpatrick’s model, learner reaction as a first level of evaluation is 
affective in nature, which can be assessed by surveys, questionnaires, focus groups, and 
interviews. What is essential is that this level is a personal reaction to the learning 
experience. Knowledge transfer is about what is actually learned. It is hard to measure 
tiiis level of learning, but it is most essential in measuring the effectiveness of the 
learnin g experience, compared to the other learning objectives. Knowledge tests are often 
present in DL courses, which enables the learner to assess his own progress and obt^ 
feedback. Another component in Kirkpatrick’s model, behavior transfer assessment, deals 
with measurable, observable behavioral changes. Unfortunately, this component is often 
not measured. Assessment methods at this level of evaluation include time-consuming 
360 degrees surveys. Organizational impact measures are usually focused on 
improvements (service response time, service quality, etc.) that increase the return on 
investment. These are not causal measures and are outside the scope of this thesis. 

Baldwin and Ford (1988) describe tr aining design as one of the main training 
input factors in the following theoretical model of the knowledge transfer process (Figure 
4). In terms of their model, this tiiesis focuses on learner characteristics and learning 
design as inputs, and learning and retention as outputs. Furthermore, relationships 
between them are of special importance for this research because the system approach by 
definition focuses on relationships between elements and subsystems inside the system, 
as well as input into and output from the system. 
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Learning Inputs Learning Outputs Conditions of Transfer 



Figure 4. Baldwin and Ford’s Model of Learning Design [From: Baldwin & Ford 

(1988), p. 65]. 


Another very interesting, and for this thesis a useful model, is proposed by Van 
Slyke, Kittner and Belanger (1998). It suggests that multiple variables must be taken into 
account concurrently in looking at their effect on distance education [Belanger & Jordan 
(2000), pp. 187-190]. Their framework targets distance education, not distributed 
learning in the sense I defined and used in this thesis. Their theoretical model addresses 
the wider spectrum of learning environments, including VTT, WBT, and others. They 
emphasize that their finmework can be used to conceptualize the learning environment 
where the learner is embedded in all its complexity. All key dimensions are present in the 
model: the institution, the learner, the course, and the type or types of distance education 
applied. Moreover, they stress that all evaluation progr ams should e 3 q)licitly define which 
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of these categories of variables are being measrired, which are being controlled, and what 
the outcome variables are. 

Belanger and Jordan go a step further, adapting Van Slyke’s model (Figure 5). 


Learner 


Institution 


Course 


DL 

Characteristics 


Characteristics 


Characteristics 


Characteristics 


FIT 



Learner 



Instructor 




Society 



Institution 


Figure 5. Framework for Distance Education and Training [From: Belanger & 

Jordan (2000), p. 189]. 

This framework, which builds on the previously mentioned models, gradually 
evolved from die analysis of learning outcomes to a very broad view including even 
society. Comparing Baldwin & Ford’s model widi Belanger and Jordan’s, we find that 
new determinants and categories of outcomes are added. Learner characteristics are 
present in both models. While Baldwin and Ford outline learning design, Belanger and 
Jordan differentiate cour^ characteristics and DL characteristics. The last learning input 
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for Baldwin and Ford is the work environment, but for the latter two authors it is 
institution characteristics and society. Belanger and Jordan (2000, p. 186) comment: 

All of these characteristics and variables must be carefully examined, not 
in isolation, but together in evaluating the appropriateness for an 
individual, a course, and an organization to attend, convert, or implement 
DL. We want to extend the model to include four difTerent units of 
analysis impacted by DL: the learner, the instructor, the institution, and the 
society. 

They also add that there are two levels of outcomes and thus two measures of 

success (or efficiency): one for institutions and one for students. As Figure 5 shows, they 

extended the number of units of analysis impacted by DL to four: the learner, the 

instructor, the institution, and society. 

C. OTHER FACTORS OF DL EVALUATION 

Phipps and Merisotis (1999), after analyzing a large amount of original studies on 

DL since 1990, concluded that most of the studies measured effectiveness of DL in one, 

less frequently two, and very rarely in three of the following areas: 

• Student outcomes, such as grades and test scores 

• Student attitudes about learning through distance education. 

• Overall student satisfaction toward distance learning 

In other words, most of the studies concentrated only on some aspects of the 

learning process and neglected other relevant parts of the system. They found serious 
methodological weaknesses in most of the studies and concluded that there was a paucity 
of original, legitimate research dedicated to ejq>laining or predicting phenomena related 
to DL. According to their review, approximately 28% of DL research studied two-way 
interactive video technologies, 28% studied computer-mediated learning, 16% studied 
one-way broadcast technologies, 13% one-way prerecorded video, 6% two-way audio 
and one-way video learning, and 9% studied other technologies. The main research 
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methods applied were: e3q)erimental 51%, descriptive 31%, case study 15%, and 
correlational 3%. 

Suchan and Crawford (1998) analyzed implementation of network-based 
instruction (NBT) to senior Navy medical executives, emphasizing in introduction the 
unique effects of technology on learner and on interaction between learner and instructor, 
as well as between learners. Their thorough analysis provided several recommendations 
at dififereijt organizational levels of a learning process. Here is the selection of 
recommendations most relevant to a course like SS3011: 

• Formal organkational arrangements: provide appropriate technology 
and training to use that technology; provide technological support. 

• Informal organization: program evaluation should be conducted at 
several points in time to account for the effects of exposure to the 
medium; use a qualitative methodology, i.e., interviews, to capture factors 
that mi ght be not anticipated by instructional designers. 

• Instructional task design factors - perceived meaningfiilness: show 
clear indication of the module’s purpose; define the module learning states 
fi:om a personal and organizational perspective; ensure that instructional 
objectives are reflected in all evaluation measures; develop an overview of 
topics to be covered to further provide learning constraints; implement 
constant summaries that rearticulate module purpose. 

• Instructional task des^ factors — perceptions of choice: create hot 
links (hypermedia) to other Web sites; monitor and feedback learners’ 
choice of information search strategies so tiiat they become more aware of 
the search patterns and structures they use; use open-ended questions 
based on readings, scenarios; encourage (students) to share their own and 
respond to other’s experiences in these electronic venues; provide 
imm ediate and direct access to a ‘real’ instructor for problems and 
feedback on tiie program. 

• Instructional task design factors - perception of competence and 
progress: provide carefully timed, ongoing feedback on progress toward 
achieving module goals; vary feedback style based on module learning 
outcomes; use positive language to fimne aU types of electronic feedback. 
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The authors conclude their recommendations with very important point diat 
feedback is, in essence, interactivity, what is of exceptional relevance for this thesis. 

Moskal and Dziuban (2000) in introduction of their paper ‘Present and Future 
Directions for Assessing Cyber-Education’ analyzed evaluation strategies from 
standpoint of the University administration. According to them, evaluation is important to 
determine both the initiative’s progression and ultimately its success. It is important, they 
explain, to track longitudinally the impact of on-line courses on students and faculty. In 
summary, they outline five assumptions of web-based teaching and learning evaluations: 

• Research should inform practice 

• Evaluation must be contextually relevant 

• Faculty must be rewarded as research leaders 

• Evaluation must be seen as an integral part of the initiative 

• As evaluation matures its demonstrable impacts must be obvious. 

The authors analyzed in detail data on both students and the faculty at UCF. 

Eighty-nine percent of web course students were very favorable about taking another 

course, and 70% of them indicated they would definitely take another on-line course. As 

main reason for enrolling on-line course, students mostly indicated convenience of use. 

The faculty teaching web courses had an increased workload, but 80.4% of them were 

satisfied or very satisfied, and 93.6% of them would probably or definitely teach another 

web-based course. They reported that the main negative on-line aspects were time 

demands (44%), technical problems (38%), and student problems, mostly with accessing 

the course content or communicating with the instructor. The authors concluded that 

‘measuring the success of on-line teaching and learning is complex procedure nested in 

series of intricate interactions’ (p.31), and that ‘outcome comparisons associated with 
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web-based learning and traditional on-campus education may be intuitively appealing, 
but they do not facilitate responsive programs.’ 

Hartman, Dziuban, and Moskal (1999) m their conclusion inducted that ‘faculty 
satisfaction and student outcomes covary when predicting success in ALN 
(Asynchronous Learning Network) programs’. This has implications for this thesis 
because SS3011 course was taught in learning environment very close to ALN as the 
authors of the article defined it, and became the instructor of the course also represented 
important source of data in this thesis research. 

Alliger, Tannenbaum, Bennet, Traver, and Shetland (1997) conducted meta¬ 
analysis of 34 studies and found utility-type reaction measures be more strongly related 
to learning than affective-type reaction measures. This finding has direct implications for 
evaluation of training effectiveness. 

Another important paper is an article by Tannenbaum, Mathieu, Salas, and 
Cannon-Bowers (1991), where authors found support for their hypotheses that tr aining 
fulfillinent, trainee reactions, and training performance are related to post-training 
organizational commitment, academic self-efficacy, physical self-efficacy, and training 
motivation. The audiors provide (p. 759) Bandura’s definition of self-efficacy as ‘the 
belief in one’s capability to perform a specific task’. The authors added that self-efficacy 
is also related to an individual’s openness to experiment and to the likelihood of using 
new technology. Another factor discussed in their article was organizational commitment. 
They defined it as the relative strength of an individual’s identification with and 
involvement in a particular organization. 
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Hill, Smith, and Mann (1987) investigated the relation between sense of efficacy 
regarding computers and people’s readiness to use them. They showed expected relation 
between efficacy beliefs with respect to computers and the likelihood of using computers. 
Also, they found previous experience with computers is related to beliefs of efficacy with 
respect to computers. However, the latter did not exert a direct independent influence on 
the decision to use computers. They concluded that their studies demonstrated die 
importance of efficacy in the decision to adopt an innovation - it is an important factor in 
determining an individual’s decision to use an iimovation. Experience per se, they say (p. 
313), does not directly affect subsequent behavior regarding further adoption of computer 
technology. Rather, experience with computer technology leads to a higher likelihood of 
technology adoption only through changes in perceived efficacy. However, because the 
students in this sample are officers of U.S. services in contemporary circumstances, they 
do not have much choice about using or not using computers; it is everyday need. 


D. STUDENT BEHAVIOR PATTERNS ON-LINE 

William A. Long's theory of reactive behavior is described in several articles: 

Long (1985), Dziuban and Dziuban (1997), and Dziuban, Moskal, and Dziuban (1999). 

In brief, it is a theory based on normal behavior patterns diat adolescents exhibit 
in a learning environment, eiqilaining how young people react to their environment and 
its associated stresses. According to Long, individuals tend to one of four behavior types, 
combined with zero to four ancillary traits. Four types of reactive behavior patterns are: 

• Aggressive Independent (AI). These students tend: to have high energy 
level and little need for peer or teacher/instructor approval; to lack 
judgment, e3q)ress their droughts and feelings impulsively; to be 
disorganized and nonlinear, preferring to work independently, often in 
leadership positions. They are challenging students, preventing 
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instructor’s complacency. Confrontation resolves their ambiguity, stress, 
or indecision. 

• Aggressive Dependent (AD). These students are high achievers, usually 
very active in student organizations, honor courses, etc. They are non- 
confrontational, eager to please, participate in class. 

• Passive Independent (PI). These students can be non-commxmicative, 
non-participatory, and stubborn. Even if having superior abilities, they 
may behave contrary to their own best interests. 

• Passive Dependent (PD). They are affectionate, sensitive, non- 
confrontational, and extremely compliant. Their need for approval is hi gh, 
dominates their relations with peers and with instructor. They see 
disagreements and criticism as personal rejection. When mature, they can 
become very caring and gentle persons. 


According to Long’s theory and many empirical findings, any of these four 
reactive behavior patterns is associated wdth none or any combination of next four 
ancillary traits (descriptions derived from D2duban, Moskal, and Dziuban [1999]): 

• Phobic Trait These students tend to develop focused fears concerning 
possible undesired outcomes of Iheir academic learning process; they are 
cautions decision makers, but excellent studaits in analytical situations. 

• Impulsive Trait They are erratic and capricious, adding vitality and 
energy to the classroom. They tend to answer a question before it is 
completely asked, or to undertake an on-line course before technical 
requirements are met 

• Obsessive-Compulsive Trait These students are thorough, methodical, 
and careful, very organized. They strongly tend to succeed in completing 
their tasks and they enjoy academic achiev^ent and excellence, but may 
lack spontaneity. - 

• Hysterical Trait These students are creative, helpful, compassionate, and 
dramatic. They may create contagious energy and enthusiasm in classroom 
environment, but in crises they become chaotic and may fail in their 
assignments. 


Long and other researchers confirmed the validity of this theory in several studies, 
includii^ studies of congruence of parents and teachers, studies of predicting types and 
traits of students from teachers’ percq)tion of fiieir academic problems, studies of the 
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correspondence of phenomenological and self-report approaches, etc. All these types and 
traits are present in normal population of students, so the theory has instructional utility 
in higher education. 

Dziuban, Moskal, and Dziuban (1999) found indications that students in on-line 
courses tended not to be independent learners. Furthermore, passive dependent students, 
who are usually rather observers than participants, reacted least favorably to on-line 
courses. The authors also suggest directions for further research and conclude that lack of 
understanding and evaluation of described types of reactive behavior patterns may 
unknowingly discourage significant segments of student population firom enrolling in 
distributed learning programs. Commenting on this finding, Dziuban and others (2000) 
concluded: ‘The virtual environment is one in which students must reevaluate their 
fundamental approach to learning’. 
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m. METHODOLOGY 


The organization of this chapter is as follows: 

• Case Analysis 

• Demographic Data & Biographies 

• Questionnaires On-Line 

• Student Tracking, Student Profiles, and Pages Tracking 

• Grades 

• Student Opinion Forms (SOFs) 

• Interviews with the Students 

• Interviews with the Instructor 

The initial idea for this thesis was triggered during preparations for the first on¬ 
line delivery of the course ‘Space Systems - Technology and Applications’ (course code 
SS3011), taught during summer quarter of academic year 2000 at the Naval Postgraduate 
School (NPS) in Monterey, California. Commander Sue Higgins, US Navy, the instructor 
of this course, was previously trained at the University of Central Florida (UCF), 
Orlando, Florida, so the resulting teaching/leaming during the course process could be 
interpreted as an application of ‘UCF-model of distributed learning’ to one course at 
NPS. Furthermore, UCF provided continuous support during the preparations for and 
during the course in hardware (servers), software (maintaining the web pages and forums, 
removing ‘bugs’, and necessary technical troubleshooting and help), and in other ways 
(occasional meetings, consultations, etc.). 

A. CASE ANALYSIS 

Case study by definition means a conceptual umbrella for multiple sub studies of 

an isolated example (Yin, 1993, p. xii), which is to say that dififerent sources of data are 
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gathered. Ultimately, this thesis focuses on describing and analyzing qualitative data, but 

the organization of the course and of this research enabled the collection of significant 

amount of quantitative data. This case study is composed of several sub-studies based on: 

o Basics on-line questionnaire 

o A-2 questionnaire 

o Module 14 questionnaire 

o Final Survey questionnaire 

o On-line tracking data 

0 Interviews with the students 

0 Interviews with the instructor 

0 Final grades 

All these sub-studies are combined and integrated with the primary and secondary 
research questions. 

The organization of the case study was dictated by the dynamics of the course and 
methods applied, i.e. the researcher had to adapt to the teaching schedule of the course to 
avoid any interference. 

The foxir evaluation instruments were put on-line at appropriate times, according 
to targeted parts of the course or modules that I wanted to evaluate. 


The instructor and I made clear the purpose of the thesis to the students on several 
occasions. The instructor introduced me to the students during the first class, so they 
could become familiar with me and with the research I planned to conduct. 

1 organized the interviews with the students individually several weeks after the 
course when exams and final papers firom their other courses were completed. I used that 
time because they were free of other academic pressures, so they could be more relaxed, 
objective, and willing to be frank and vocal driring the interviews. 
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Finally, I interviewed the instructor: before, during, and after the course. These 
latter occasions were much more directed by my observations, by questions that appeared 
fi’om other data, or by her opinion of what else was important but still not mentioned. 

Some of the instruments used in this research called for anonymous responses, 
and some were associated with student identity (names). Consequently, it was possible to 
explore relations between variables firom different instruments (qualitatively and 
quantitatively), including measures of association (for instance coefficient of rank 
correlation). Such identifiable sources of data in this research were: 

o The Basics - Demographics and Studying Strategies (A-1) on-line 
questionnaire 

o Module 14 Evaluation 

o Biographies (all except two students posted them to forum, as the 
instructor requested in order to enable the students to get know 
each other) 

o Student Tracking, Student Profiles, and Pages Tracking. WebCT 
software was set to automatically track for each student the number 
of visits to web pages, number of read articles in forums, number 
of posts to forums, and additional data. 

o Grades 

o Individual interviews 

It was not possible to associate other questionnaires (After-orientation 

Questionnaire (A-2) and Final Survey) with student identity, so analysis of responses 

may only be made in the distribution of the answers, of associations of variables within 
the instruments, and in qualitative analysis. One of the main reasons for this limitati on 
was the way the WebCT software was set. As measures of association, I used two 
different coefficients of correlation, depending on whedier a variable was numeric or 
ordered categorically: 
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Speamian’s Rho (p) coefiScient of rank correlation is applicable with ordered 
categorical variables and with a given sample size. Some other indicators (based on 
and some others) could be used too. The equation for calculating the Rho coefficient is 
shown in Equation 1. 
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Equation 1. Spearman’s Rho Coefficient of Rank Correlation [From: Conover (1999), 

p.315]. 

For niimerical variables, Pearson’s coefficient of correlations was calculated: 
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Equation 2. Pearson’s r Coefficient of Correlation [From: Berk & Carrey (2000), p. 

321]. 

For both coefficients I provided respective p-values. In calculating p-value for 
each of the coefficients, the null-hypothesis (Ho) is that variables in the questions are 
mutually independent, while alternative hypothesis (Ha) finds a tendency either for the 
larger values of Xi to be paired with the larger values of Yi, or a tendency for smaller 
values of Xj to be paired with the larger values of Yi in which case, the coefficient would 
have negative sign. Due to the small sample in this research, a p-value of 0.05 would be 
acceptable as statistically significant. 
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B. DEMOGRAPHIC DATA 


Only selected demographic data were collected, mostly within the fiorst on-line 
questionnaire (see next section). Not all data from this set is applicable to this thesis, so 
part of these data is reported in Appendix F. Distributions and other relations are 
explored in Chapter IV. 

C. QUESTIONNAIRES ON LINE 

Thirty-two students attended SS3011, but only thirty of them completed it as 
regular NPS students: two of them were not regular NPS students and they were not 
getting credit for finishing the course. Moreover, they were off campus as non-resident 
additional attendees of the course and were treated differently from the other students. 
For these reasons, they are excluded from the sample. 

Four on-line questionnaires were administered to the students (Appendices A 
through D). The instructor’s support staff from the Tr ainin g System Division, Naval Air 
Warfare Center, Orlando, Florida, provided the initial set of questions, but many items 
were adapted and modified. Also, a significant number of new questions were created 
because of the needs of the instructor and the research questions for this thesis. A new 
instrument was incorporated into the last questionnaire: Long-Dziuban’s instrument with 
slightly modified instructions. The authors approved its application in this thesis. The 
four questionnaires were: 

• The Basics - Demographics and Studying Strategies (A-1). This 
instrument covered general facts about the students, but focused mostly on 
what the instructor was interested in order to know her students better. 
Besides questions about e-mail address, student status, service, pay grade 
and MOS, there are also questions about previous experiences relevant for 
attending the course (former space-related duties or courses attended, 
experience in using web resources), about expectations (expected nvunber 
of web visits weekly, overall expectation on the on-line course), self- 
perceived proficiency in using computers (proficiency in using computers. 
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on-line research techniques), and concern about one’s own ability to use 
the web-based materials in the class. The questionnaire also provided the 
students an opportunity to make comments in their own words. It was put 
online at the end of first module. It had a total of 25 items. 

• After-orientation Questionnaire (A-2). This instrument contained 26 items. 
It was directed towards the students’ comfort level on various aspects of 
the course, and was administered after the first two weeks of the course. 
For instance, the students were asked about setting-up their systems, 
accessing the Internet, word processing, posting to forums, sending e-mail, 
using WebCT software, on technical support, and so on. Also, they had 
several items where they could respond in their own words and comment 
on different issues about their first two weeks of the course. Several 
questions explicitly asked them about their comfort level with the most 
important course features, like forums, e-mail, quizzes, links, and others. 

• Module 14 Evaluation was applied after the fourteenth module. Module 14 
was especially interesting for two reasons. First, it was different from 
other modifies, and therefore, deserved special attention. Second, that 
point in time was close to the end of the course, so it was a very good 
moment to ask the students overall questions about the course. This 
questionnaire was the shortest (only 14 items), and there were three main 
groups of questions: 

o Questions directly related to previously specified learning 
objectives (about abilities to identify key elements in Space 
Control, abilities to synthesize others’ findings in forums, 
readiness to interact with subject matter expert in the area, etc.), 
o Questions about final comfort level in accessing the Internet, 
sending e-mail, and posting comments to forums, 
o Open-ended question to provide comments and additional 
thoughts. 

• The Final Survey was applied after the final exam/paper, but before the 
students received their final grades for the course. It had 34 questions and 
contained the following groups of items: 

o Questions the instructor foimd very interested, especially in the 
context of the ultimate learning objectives of the course, 
o Questions about the perceived usefulness of several features used 
during the course (guidelines at the course homepage, quizzes, 
module readings, textbook readings, integration of responses to 
forums). 

o Questions about affective reactions (liked/disliked) to several 
features. 

o Open-ended questions about the instructional strategies and 
materials, usability of everything they learned in the course, 
flexibility, fundamental changes in the approach to learning, and 
similar topics. 
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o Four questions about tiie interactivity as an important aspect of the 
learning process. 

o Long-Dziuban’s on-line reactive patterns instrument 
Though it may seem that four on-line questionnaires during the course took too 

much of the students’ time, data about the time the students actually spent in answering 

the questionnaires clearly show that it was not the case. At most they needed between 

three and five minutes per questionnaire, and the majority of them chose to answer all or 

almost all of the questiotmaires. 

Thirty of the students answered the first questionnaire; thirty-one the second one; 
twenty-six the third questionnaire; twenty-one answered the Final Survey questionnaire. 


D. STUDENT TRACKING, STUDENT PROFILES, AND PAGES TRACKING 
WebCT software had installed a system of tracking student on-line activities: 

o Total number of visits to the course pages (also called total number 
of hits) 

o Total number of visits to the course homepage 

o Total number of visits to the course content page 

o Total number of forum articles (posts) read 
0 Total number of forum articles (posts) posted 
o Total number of original forum articles posted (i.e. not as 
responses to someone’s other posts) 

o Total niunber of follow-up forum articles posted (i.e. as responses 
to someone’s other posts) 

o Date and time of posting (for each forum post) 

o Number of different course pages visited 

E. STUDENT OPINION FORM (SOFS) 

Student Opinion Forms are the standardized and mandatory formal way of 

obtaining feedback for any course taught at NPS. They are being applied not only in 
order to evaluate student perception of instructor and course effectiveness, but also as an 
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additional help and support to the instructor’s eiforts in improving the course, the 
methods of instruction, and other important aspects of teaching. 

SOFs are always applied anonymously on forms that an equipped computerized 
system can read. The current version of SOF forms in use is NPS 5040/2 (REV 3-84)]. 
Besides elementary data about the course, on the student’s curriculum, the number of 
hours during the quarter the student attended, and the number of quarters already 
completed, there were 16 questions about the course, the quality of teaching, the exams, 
the textbook, and similar topics. Students answer all 16 questions on a scale from 5 to 0 
(from ‘strongly agree’ to ‘strongly disagree, including a category for ‘no comment’). It 
was easy to quantitatively analyze student feedback, but an appropriate qualitative 
analysis of student responses can also lead to qualitative conclusions about the overall 
success of the course. 

F. GRADES 

The instructor provided a list of final grades for all SS3011 students who 
successfully completed the course. The analyses include these data. 

G. INTERVIEWS WITH THE STUDENTS 

In the beginning of the course, the instructor introduced me to the students and 

explained the goals of this research. Near the end of the course, the instructor sent an e- 
mail to the students announcing I woiild like to conduct brief, post-course individual 
interviews with the students who voluntarily chose to participate. Then, I sent individual 
e-mail messages to each of them. 

I prepared eleven open-ended questions (Appendix E). As a clinical psychologist 

with experience in himdreds of individual interviews, questionnaires, and other research 

methods with military officers and enlisted personnel, I planned to keep the interviews 
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focused on questions and to intervene minimally, only when it was necessary to redirect 
their comments and explanations back to the questions. I intended to encourage the 
students to talk J&ankly and to feel free to give critiques, recommendations, and 
suggestions concerning the on-line course. 

The most interesting and intriguing responses and comments from the four on-line 
questionnaires, and dilemmas I expected in the discussion of the research questions 
primarily inspired the choice of these interview questions. 

I decided to conduct the interviews one to two months after the course was 
completed so they would not feel pressure from otiier final exams or written assignments 
and be more or less free from the ‘fresh’ impressions of their final grade in SS3011. 

H. INTERVIEWS WITH THE INSTRUCTOR 

I had over fifteen formal and informal meetings with the instructor. I listed areas 

and prepared questions, which I then I sent to her several days before the arranged 
interviews, for the purpose of direct collection of facts, her opinions, interpretations, and 
comments on relevant course issues. The interviews were conducted without a third party 
present and in a pleasant office atmosphere. I took notes as the instructor answered the 
questions and added comments. 
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IV. RESULTS AND DATA ANALYSIS 


In this chapter I provide data I collected using all instruments in this research: 
four on-line questionnaires, Student Opinion Forms, student on-line tracking, final 
grades, and interviews with the students and with the instructor. Besides that, I made the 
appropriate data analysis, mostly concentrating on three key points: distributions of the 
responses/data, degree of association between different variables (in terms of coefficients 
of correlation or rank correlation), and qualitative analysis of responses to open-ended 
questions or interview questions. Data not relevant to research questions are omitted or 
moved to Appendix F. After each section, I provide a summary of the data analysis in the 
respective section and a brief discussion. Integration of all results and overall discussion 
appears in Chapter V. 

A. QUESTIONNAIRES ON-LINE 

1. First questionnaire — ‘THE BASICS, Your Demographics and 
Studying Strategies’ 

The first question (Appendix A) was about student e-mail addresses, and secondly 
about their status as a NFS student. All students were full-time NFS students. The 
number of valid responses was thirty. 

Table 1 shows students by service. 


Service 

Number of students 

Percent (%) 

US Marine Corps 

13 

43.34 

US Navy 

10 

33.33 


4 

13.33 

us Air Force 

0 

0.00 

Others 

3 

10.00 

Total 

30 

100.00 


Table 1. Students in the Sample by Service. 
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Thirteen officers (43.33%) were Marine Corps officers, one third were Navy 
officers, and three were US Army officers . There were no Air Force officers, and the 
service of the others was not specified. 


Table 2 below shows the distribution of the students by pay grade (not rank). 


Pay Grade 

Number of Students 

Percent (%) 

0-3 

18 

60.00 

0-4 

10 

33.33 

0-5 

1 

3.33 

Not available 

1 

3.33 

Total 

30 

100.00 


Table 2. Students in the Sample by Pay Grade. 


Since pay grade is an ordered categorical category (except where datum on pay 
grade was not available), it is possible to check its possible associations with other 
numerical or ordered categorical variables - not only with such variables in the first 
questionnaire, but also in the Module 14 questionnaire (which was also identifiable by 
student’s name) and in student on-line fracking variables. 

Statistical indicators like the mean or standard deviation would not be appropriate 
here because responses to these statements are ordered as categorical variables where 
semantic distances between categories are only artificially assumed to be equal. Only 
nonparametric statistical methods are applicable because the assmnption about normal 
distribution does not have any justification in this situation. Also, the sample is very 
small. 

For pay grade, only one rank correlation coefficient was statistically significant 
(Table 3). 


36 


















Instrument 

Statement 

P 

p value 

First 

questionnaire 

‘Have you experienced an on-line course (or 
courses) that used a web site to do/complete 
coursework?’ 

0.404 

0.027 


Table 3. Significant Correlation and Level of Significance between Student Pay 

Grade and Another Variable. 

Students with a higher pay grade (i.e. with higher rank) tended to have 
significantly more pre'vdous on-line experience. 


The eighth item the students were asked was about the number of quarters they 
had been enrolled, including the current quarter. Their responses are analyzed in Table 4. 


Quarters completed 

Number of students 

Percent (%) 

None 

12 

3.33 

One 

3 

10.00 

Four 

17 

56.67 

Five 

8 

26.67 

Six 

1 

3.33 

Total 

30 

100.00 


Table 4. Students in the Sample by Number of Quarters Enrolled (Including 

Current Quarter). 

Three student tracking variables had significant correlations with the number of 
quarters enrolled. Since all these variables were numerical, I used Pearson’s r coefficient 
of correlation. Table 5 shows significant correlations. 


^ Tliis student probably didn’t understand the question, because it asked for number of quarters 
enrolled including current quarter. Other data showed that all students in die sample had status of regular 
NFS student. 
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Instrument 

Variable 

r 

p value 

Student tracking 

Number of visits to the course homepage. 

-0.479 

0.007 

Student tracking 

Number of posts to the forums. 

-0.432 

0.017 

Student tracking 

Number of follow-up posts to the forums. 

-0.388 

0.034 


Table 5. Significant Correlations and Levels of Significance between the Number 
of Quarters Enrolled and Other Variables (Ordered by Significance). 

Senior NFS students (those enrolled in their fourth to the sixth quarter) slightly, 
but statistically significantly, tended to visit the course homepage fewer times, to post 
fewer forum articles, and fewer follow-up posts to the forums. 

Asked about previous space tours/duties, only two students responded they had 
‘some training’. One responded he had several programs, and all other students none. 

The next question was about previously attending any other space-related 
course(s). Only one student had attended one such course, and all the others none. 

Asked about previously attending any on-line coiurses, only three students 
responded they had such experience. One of them was actually attending an MBA 
program on-line. 

The eighteenth item was about the expected fi'equency of using the course web 
site. The responses are shown in Table 6. 


Expected frequency 

Number of students 

Percent (%) 

3-5 times per week 

23 

76.66 

2 times a week 

2 

6.67 

At least once a week 

2 

6.67 

‘I don’t know’ 

3 

10.00 

Total 

30 

100.00 


Table 6. Expected Frequency of Using the Course Web Site. 
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This variable has significant correlation to only one variable: the number of 
original postings made by the student in forums (student tracking). Spearman’s p is 
0.423, p is 0.020. The students widi a higher than expected frequency of accesses tended 
to send more postings to forums during the course. 


The next item the students were asked was about where they plan to use access to 
the course the most. Table 7 shows the responses. This is a non-ordered categorical 
variable, so rank correlation is not applicable. One half of the students planned to access 
the course from their homes, which indicates they intended to use the convenience of 
accessing the on-line course from a place other than the school. 


Location 

Number of Students 

Percent (%) 

BOQ room 

0 

0.00 

A house 

153 

50.00 

Computer lab in campus 

14 

46.67 


1 . 

3.33 

Shared area in living quarters 

0 

0.00 

Others 

0 

0.00 

Total 

30 

100.00 


Table 7. Location that Students Expect to Use the Most to Access the Course. 

The next question asked the students about their self-perceived level of computer 
proficiency: ‘On a scale of 1 to 5, where 1 = novice and 5 = expert, how would you rate 
your current proficiency in using computers’? Figure 6 shows the distribution of the 
responses. Most of the students (seventeen or 56.67%) perceived themselves at a 
proficiency level between ‘medium’ and ‘expert’. Only one student chose a response 
between ‘novice’ and ‘medium’. This variable showed statistically significant correlation 
with three other variables (Table 8). 


^ One student added, answering the following question - ‘possibly at school in one of the labs’. 
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Figure 6. Distribution of Self-Perceived Computer Proficiency. 

N = 30 Mean = 3.77 Median = 4 

Minimum = 2 Maximum = 5 Mode = 4 



Statement 

P 

P value 

First 

questionnaire 

‘On a scale of 1 to 5 where l=novice and 
5=expert, how would you rate your on-line search 
resources and techniques?’ 

0.836 

< 0.001 

First 

questionnaire 

‘Please indicate the extent to which you agree 
with the following statement: 

I expect this online course will be valuable and 
improve my learning:’ [l=strongly disagree ... 
5=strongly agree] 

-0.498 

0,005 


Number of articles read 

0.425 

0.019 


Table 8. Significant Correlations and Levels of Significance between Self- 
Perceived Computer Proficiency and Other Variables (Ordered by Significance). 

Students who perceived themselves as more computer proficient strongly tended 
to perceive themselves as better in using on-line search resources and techniques. The 
second relation (a much weaker correlation than the first one) means that students who 
perceive themselves as more computer proficient expected to gain and learn less from 
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this on-line course than students who perceive themselves as not very computer 
proficient. The third relation shows that the students who perceived themselves as more 
computer proficient tended to read more articles in the course forums. They probably 
found the forums easier, more comfortable, less distractible, could pay more attention to 
the articles, and/or tend to be more active in reading the forums articles. 

The next question in the first questionnaire was about self-perceived on-line 
searching skills: ‘On a scale of 1 to 5 where l=novice and 5=expert, how would you rate 
your on-line search resources and techniques?’ Figure 7 shows the distribution of the 
responses. Fourteen students (46.67%) saw themselves as between ‘medium’ and 
‘expert’, nine students (30.00%) as ‘medium’, while only four students (13.33%) thought 
they were between ‘novice’ and ‘medium’ in search techniques. 

Self-Percieved Search Techniques 


Number of 
students 


Novice Medium Expert 

0 


Figure 7. Distribution of Self-Perceived On-Line Search Skills. 

N = 30 Mean = 3.63 Median = 4 

Minimum = 2 Maximum = 5 Mode = 4 

Five correlations with other variables were significant (Table 9). 
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Instrument 

Statement 

P 

p value 

First 

questionnaire 

‘On a scale of 1 to 5, where 1 = novice and 5 = 
expert, how would you rate your current 
proficiency in using computers.’ 

0.836 


Student tracking 

Number of different pages visited 

-0.404 

0.027 

Module 14 

‘Identifying elements of space systems that are 
vulnerable to disruption, degradation or 
destruction by adversaries.’ [1 = strongly 
disagree ... 7 = strongly agree] 

0.398 

0.030 

First 

questionnaire 

‘Please indicate the extent to which you agree 
with the following statement: 

‘I expect this online course will be valuable 
and improve my learning’ [l=strongly disagree 
... 5=strongly agree] 

-0.368 

0.045 

Module 14 

‘Evaluating the likelihood that various types of 
space coimtermeasures might be used across 
the peace-war continuum.’ [1 = strongly 
disagree ... 7 = strongly agree] 

0.367 

0.046 


Table 9. Significant Correlations and Levels of Significance between Self- 
Perceived On-Line Search Skills and Other Variables (Ordered by Level of Significance). 


The first relation, the only high correlation to other variables, means that students 
who saw themselves as more computer proficient strongly tended to use on-line search 
techniques better. The second relation is also logical: students who think of themselves as 
skilled in on-line search techniques expect to gain less firom the course (in terms of 
valuable improvements in their learning), while students with less on-line experience and 
skill expect to gain more because of the on-line nature of the course. 

The relations with items firom the Module 14 questionnaire are hard to explain. 
Possibly, students with high on-line search skills (by their own assessment) later in the 
course achieved more in the sense of the course learning objective described in the 
respective question. It is interesting that the students with higher level search skills 
tended to visit a smaller number of course pages. A possible explanation would be that 
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they were able to get to pages they wanted to visit easier, as opposed to students less 
skilled in search techniques, who had to use the ‘longer way’ and thus visited more 
different pages in order to get what they wanted from the course web site. 

The next scale was about the self-perceived expectations of improvement that the 
comse would provide in terms of their learning techniques: ‘Please indicate the extent to 
which you agree with the following statement: I expect this on-line course will be 
valuable and improve my learning.’ Figure 8 shows the distribution of the responses. 

Over 73% students expected the on-line course to improve their learning 
(‘somewhat’ or ‘strongly’), and only two of them disagreed with that statement. This 
variable had five significant correlations (Table 10). Correlations with other variables in 
this questionnaire are telling us that more experienced and skilled students expect to gain 
less from the course. The only conclusion to make about the other three correlations is 
that expectations on learning improvement are somewhat associated with self-assessed 
achievements in meeting learning objectives of Module 14. 


"I Expect On-line Course to Improve my 
Learning” 



Figure 8. Distribution of Expected Learning Improvement. 
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N = 30 
Minimum = 1 


Mean = 3.97 
Maximum = 5 


Median = 4 
Mode = 4 


Instrument 

Statement 

P 

p value 

First 

questionnaire 

‘How would you rate your current proficiency in 
using computers?’ 

-0.498 

0.005 

Module 14 

‘Interacting with a subject-matter expert whose 
knowledge you can use to solve space-system 
problems.’ 

0.452 

0.012 

Module 14 

‘Identifying key elements of Space Control.’ 

0.405 

0.026 

Module 14 

‘Participating in forums (e.g. discussing, 
identifying/ consulting subject matter experts), to 
evaluate space control issues.’ 

0.392 

0.032 

First 

questioimaire 

‘How would you rate your on-line search resources 
and techniques?’ 

-0.368 

0.045 


Table 10. Significant Conrelations and Levels of Significance between Expected 
Learning Improvement and Other Variables (Ordered by Level of Significance). 

The next item asked students about their concerns about the course: ‘I am 
concerned about my ability to use the web-based materials in this class’. The distribution 
of the responses is shown in Figure 9. This question was set in an inverted direction: 
more desirable responses are on the left side of the scale. Sixteen students (53.33%) 
strongly disagreed with the statements, i.e. they were not concerned about their abilities 
to use the web-based materials at all. However, five students (16.67%) stated they were 
‘somewhat concerned’, and eight (26.67%) chose neutral response. 


Only three correlations between this statement and other variables were 

significant (Table 11). I expected the first two relations: students who are more proficient 

in using computers and more skilled in searching on-line are less concerned about their 

own abilities of using web materials in the course. The third correlation shows that more 

concerned students saw Module 14 material less useful to them in attaining high 

achievement in the forums discussions on space control issues. The question asked 
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students how the material presented in their module enabled them to achieve the 
respective learning objectives. 



Figure 9. Distribution of Stated Concerns about Own Ability to Use Web Materials 

in the Course. 

N = 30 Mean = 1.97 Median = 1 

Minimum = 1 Maximum = 4 Mode = 1 


Instrument 

Statement 

P 

p value 

First 

questionnaire 

‘On a scale of 1 to 5, ^\^ere 1 = novice and 5 = 
expert, how would you rate your current proficiency 
in using computers.’ 

-0.498 

0.005 

First 

questionnaire 

‘On a scale of 1 to 5 where l=novice and 5=expert, 
how would you rate your on-line search resources 
and techniques?’ 

-0.368 

0.045 

Module 14 

‘Participating in forums (e.g. discussing, identitying/ 
consulting subject matter experts), to evaluate space 

-0.364 

0.048 


_I _ control issues.’ _|_ J_ 

Table 11. Significant Correlations and Levels of Significance between Concerns on 

Own Ability to Use Web Material in the Course and Other Variables (Ordered by Level 

of Significance). 
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This questionnaire ended with an open-ended opportunity for the students to add 


any comment: ‘Any additional comments, concerns, or suggestions are welcome.’ 


Sixteen students (53.33%) added comments. Analyzing their responses, I grouped 
them into four typical kinds of comments: Positive first impressions, advice and 
recommendations, neutral comments, and concerns'. Here are their responses: 

Positive first impressions 

• I like what I’ve seen so far! 

• None for now other than site seems to be very user friendly and well 
developed. 

• Looking forward to this quarter and learning environment. 

• I’m looking forward to trying an on-line course! 

• I was a bit skeptical, but so far the course seems to be going well. The 
thing I like best is the flexibility -1 can read and take the quizzes when it 
is convenient for me. I worried that I might not keep up, but that has not 
been a problem. 

• Great response times to technical problems. The format is very well 
designed and implemented. 

Advice and recommendations 

• Class sizes should be smaller or groups should be assigned earlier. The 
general forum is too large. 

Neutral comments 

• None, at this time. Looking forward to this first time experiment with use 
of online learning. 

• None at this time. Good Survey! 

• I have no idea how much time this class will take. 

• I do prefer traditional in class teaching. Online course work requires more 
effort because of the all the keystroke required to complete a course. 
Verbal communications allows the student to quickly ask a question and 
get an answer. Spell checking is needed on these web pages to help speed 
up and reduce keystrokes. 
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As this is my first web-based course, I am apprehensive. I prefer a more 
formal lecture style of learning. However, I will give this comse my best 
effort. 


Concerns 

• I’m worried I’ll spend more time trying to figure out fius program then 
learning about space. 

• lam concerned about potential computer and Internet related problems. 

• That website access will be available when required and I won't have to 
continuously fight the system to get access is a concern. 

• It seems that the server is having some capacity issues at peak times, i.e. 
during school hours. More often than not I am getting a ‘page not 
available’ image when I click on links or postings. Makes filings 
aggravating, but not impossible. I can live with it, but those that chose #1 
for questions #21 and #22 might be hurting.^ 


There were more positive and neutral responses than negative comments. There 
were also some other ‘signals’ fi:om the students that deserve additional attention: 
comments that some of them personally simply prefer classroom (face-to-face) learning 
to on-line learning; that their concerns are about technology malfunctions, not about the 
DL course itself, and that class size and forums could be too large. 


First questionnaire - Summary 

This on-line questionnaire was designed to collect some general facts about the 
students and to collect data the instructor and I wanted to know firom this group of 
students. 

Thirty students completed the questionnaire. The questionnaire collected their e- 
mail addresses, data on their status as NFS students, their service (over 43% were 

^ Answer #1 to questions 21 and 22 are students who identified themselves as novices in computer 
proficiency and in on-line search techniques. In this sample there were no such students. 
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Marines, one third were Navy officers, four were US Army, and three others), and pay 
grade (60% had 0-3, one third 0-4, and one 0-5). Pay grade had significant rank 
correlation with previously experienced on-line course or courses, i.e., officers with a 
higher pay grade (higher rank) tended to have more experience with on-line courses. 
However, this was only a mild correlation. 

The number of enrolled quarters varied from none to six, with a mean of 3.9. This 
variable had no significant correlations with any other variables, except with three 
student tracking variables: NPS students attending later quarters tended to visit the course 
homepage fewer times, to send fewer posts to forums, and fewer follow up posts to the 
forums. 

Only two students had some previous space-related duties, and only one had taken 
previous space-related course. Three students had experience with an on-line course, 
including one who attended an MBA program on-line. Approximately half the students 
responded they would access the course web sites mostly from home, and another half 
mostly from computer lab in the campus. 

Four five-point scales about self-perceived computer proficiency, on-line search 

skills, expectations to improve learning, and concerns about own abilities to use web- 

based materials in the course had some interesting correlations with each other, 

correlations vdth some data from on-line student tracking, and with several items from 

the Module 14 questionnaire. Between two and five students thought they had low 

computer proficiency, limited on-line search skills, or they had significant concerns about 

their own abilities to use web-based materials. Responses to the statement T expect this 

online course will be valuable and improve my learning’ had negative correlation with 
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some other variables: the more a person knows about computers and the Internet, the less 
he/she expects to improve his/her learning in the course. It is important to note that 
‘improving one’s own learning’ is not same as ‘leam in a course’: the first refers to 
learning skills and habits, and the second to knowledge and skills in a subject matter. 

Several tendencies can be concluded from the student comments: their feedback 
was more positive and neutral than negative; some of them just preferred a traditional 
classroom to an on-line course (without explicit explanation). The student concerns were 
more about technical problems than about the DL course itself. Some complained about 
the forums being too large. 

2. Second Questionnaire - ‘A>2’ 

This questioimaire (Appendix B) was anonymous. It was administered after the 
orientation part of the course, i.e., during the second and part of the third week of the 
course. Thirty-one students completed the A-2 on-line questiormaire, which means that 
besides resident NPS students, one or two of the non-resident students responded to the 
questiormaire. From a total of 26 questions/statements, 17 were in the form of five-point 
scales (from ‘Extremely uncomfortable’ to ‘Extremely comfortable’), and the remaining 
questions were open-ended. 

The first question was ‘How would you rate your comfort level at this point with 
setting-up your system?’ Figure 10 shows the distribution of the responses. 

Clearly, majority of the students (twenty-seven out of thirty-one or 87.01%) did 
not have significant problems with setting-up their PC. Only four students said they were 
somewhat imcomfortable. 
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There were eight statistically significant rank correlations with other variables 


(Table 12). 


Comfort Level with Setting-up Own System 



Figure 10. Distribution of Responses on the Comfort Level with Setting-up Own 

System. 

N = 31 Mean = 4.06 Median = 4 

Minimum = 2 Maximum = 5 Mode = 4 

The students who tended to have a lower comfort level widi their PCs tended to 
have a lower comfort level with other technically demanding aspects of the course: using 
forums, posting to forums, sending e-mail, using word processing and accessing the 
Internet. Also, they tended to see some of the features the instructor provided the students 
to help them with the learning process as less effective than the others: ‘Rules of 
engagement’, course overview and coiurse syllabus. ‘Rules of engagement’ was hypertext 
linked to the course homepage, with clearly stated expectations and standards the 
instructor had about student professionalism, course participation, studying, initiative, 
and academic level of communication. The course overview, written in the same media. 
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briefly defined the objectives and organization of the course. The content of the course 
syllabus (also written in hypertext) was self-explanatory. 


Question 

P 

p value 

‘How would you rate your comfort level at this point with 
posting to the forum?’ 

0.692 

<0.001 

‘How would you rate your comfort level at this point with 
the forum feature being used to teach this course?’ 

0.614 

<0.001 

‘How would you rate your comfort level at this point with 
sending electronic mail?’ 

0.526 

0.002 

‘How effective were the rules of engagement? That is, to 
what extent did the rules of engagement help you to 
imderstand what would be involved in taking the Space 
Systems Technologies and Applications on-line course?’ 

0.441 

0.013 

‘How effective was the coiurse overview? That is, to what 
extent did the course overview help you to understand 
what you would learn in the Space Systems Technologies 
and Applications on-line course?’ 

0.421 

0.018 

‘How would you rate your comfort level at this point with 
word processing?’ 

0.409 

0.022 

‘How effective was the course syllabus? That is, to what 
extent did the course syllabus help you to imderstand what 
you would learn in the Space Systems Technologies and 
Applications on-line course?’ 

0.401 

0.026 

‘How would you rate your comfort level at this point with 
accessing the Internet?’ 

0.400 

0.026 


Table 12. Significant Correlations and Levels of Significance between the Comfort 
Level in Setting-up Own System and Other Variables (Ordered by Level of Significance). 

The second question was: ‘How would you rate your comfort level at this point 
with accessing the Internet?’ The distribution of student responses is in Figure 11. 
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Comfort Level with Accessing the internet 



Figure 11. Distribution of Responses on the Comfort Level with Accessing the 

Internet. 

N = 31 Mean = 4.68 Median = 5 

Minimum = 1 Maximum = 5 Mode = 5 

Clearly, the majority of the students (25 students or 80.65% responded ‘extremely 
comfortable’) were very comfortable in accessing the Internet Only one had major 
problems, and six were more ‘somewhat’ than ‘extremely’ comfortable. Obviously, the 
students had relatively more problems in setting up their systems and their PCs than with 
getting on-line. 

This variable showed statistically significant correlations with twelve out of 
sixteen other ordered categorical variables in the questionnaire (Table 13). 

The highest correlation this variable had is, surprisingly, with the comfort level in 
word processing. The students who easily used word processing strongly tended to easily 
access the Internet. The second highest correlation is the association with the comfort 
level in using the links feature in the course. Other correlations are not so high, but they 
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certainly indicate that the comfort level in accessing the Internet is very indicative of the 
overall comfort level a student expressed about different aspects of the DL course. 


Question 

P 

p value 

‘How would you rate your comfort level at this point with 
word processing?’ 

0.737 

<0.001 

‘How would you rate your comfort level at this point with the 
links (i.e., additional web sites) feature being used to teach 
this course?’ 

0.728 

<0.001 

‘How would you rate your comfort level at this point with the 
forum feature being used to teach this coxirse?’ 

0.587 

0.001 

‘How effective was the course overview? That is, to what 
extent did the course overview help you to understand what 
you would learn in the Space Systems Technologies and 
Applications on-line course?’ 

0.581 

0.001 

‘How would you rate your comfort level at this point with 
posting to the forum?’ 

0.571 

0.001 

‘How would you rate your comfort level at this point with 
sending electronic mail?’ 

0.529 

0.002 

‘How would you rate your comfort level at this point with the 
email feature being used to teach this course?’ 

0.477 

0.007 

‘How effective were the rules of engagement? That is, to what 
extent did the rules of engagement help you to understand 
what would be involved in taking the Space Systems 
Technologies and Applications on-line course?’ 

0.451 

0.011 

‘How effective were module objectives? That is, to what 
extent did the module objectives help you to imderstand what 
you would learn in the Space Systems Technologies and 
Applications on-line course?’ 

0.406 

0.023 

‘How would you rate your comfort level at this point with 
setting-up your system?’ 

0.400 

0.026 

‘How would you rate your comfort level at this point with 
quiz feature being used to teach this course?’ 

0.399 

0.026 

‘How effective was the course syllabus? That is, to what 
extent did the course syllabus help you to understand what 
you would learn in the Space Systems Technologies and 
Applications on-line course?’ 

' 0.398 

0.027 


Table 13. Significant Correlations and Levels of Significance between the Comfort 

Level in Accessing the Internet and Other Variables (Ordered by Level of Significance). 
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The word processing comfort level was the next question: ‘How would you rate 
your comfort level at this point with word processing?’ Figure 12 shows the distribution 
of the responses. 


Comfort Level with Word Processing 



Figure 12. Distribution of Responses on the Comfort Level with Word Processing. 

N = 31 Mean = 4.65 Median = 5 

Minimum = 1 Maximum = 5 Mode = 5 

Almost all students were very comfortable with word processing (85.65% 
responded ‘extremely comfortable’). This is understandable because of the common 
usage of word processing in modem office work. Only two students had significant 
problems with word processing. 

The word processing comfort level variable was significantly correlated with ten 
other variables in the questionnaire (Table 14). Similar to accessing the Internet comfort 
level variable, this variable seems to be very indicative of the overall comfort level with 
DL of a student in this course. The highest correlations are with the comfort level in 
sending electronic messages and with the comfort level in accessing the Internet. These 
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three variables are probably key indicators of how well a person is able to handle this 
kind of course in terms of the comfort level. Further analysis will show if there are any 
other variables with similar significance in this instrument. 


Question 

_P_ 

p value 

‘How would you rate your comfort level at this point with 
sending electronic mail? • 



‘How would you rate your comfort level at this point with 
accessing the Internet?’ 

0.737 

< 0.001 

‘How would you rate your comfort level at this point with 
the links (i.e., additional web sites) feature being used to 
teach this course?’ 

0.541 

0.002 

‘How would you rate your comfort level at this point with 
posting to the forum?’ 

0.560 

0.004 

‘How would you rate your comfort level at this point with 
the forum feature being used to teach this course?’ 



‘How effective were the rules of engagement? That is, to 
what extent did the rules of engagement help you to 
understand what would be involved in taking the Space 
Systems Technologies and Applications on-line course?’ 

0.450 

0.011 

‘How would you rate your comfort level at this point with 
the email feature being used to teach this course?’ 

0.425 

0.017 

‘How effective was the course overview? That is, to what 
extent did the course overview help you to understand 
what you would learn in the Space Systems Technologies 
and Applications on-line course?’ 

0.425 

0.017 

‘How would you rate your comfort level at this point with 
setting-up your system?’ 

0.409 

0.022 

‘How would you rate your comfort level at this point with 
quiz feature being used to teach this course?’ 

0.388 

0.031 


Table 14. Significant Correlations and Levels of Significance between the Comfort 

Level with Word Processing and Other Variables (Ordered by Level of Significance). 

The fourth question addressed the forums: ‘How would you rate your comfort 
level at this point with posting to die forum?’ Figure 13 shows the distribution of the 
responses. 
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Figure 13. Distribution of Responses on the Comfort Level with Posting to the 

Forums. 

N = 31 Mean = 4.00 Median = 4 

Minimum = 1 Maximum = 5 Mode = 4 

A large majority (26 students or 83.87%) of students felt comfortable with posting 
to the forums, but more of them were ‘somewhat comfortable’ (16 students or 51.61%) 
than ‘extremely comfortable’ (10 students or 32.26%). Four students (12.90%) felt 
uncomfortable, and one was neutral about this issue. 

Table 15 shows significant correlations. Two variables about the comfort level 
with two aspects of the forum used in the course are very highly correlated. Other 
correlations confirm that this variable is one of those central comfort level variables 
within a DL environment. 

The fifth question was: ‘How would you rate your comfort level at this point with 
sending electronic mail?’ Figure 14 shows the distribution. 
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Question 

P 

p value 

‘How would you rate your comfort level at this point with 
the forum feature being used to teach this course?’ 

0.800 

< 0.001 

‘How would you rate your comfort level at this point with 
setting-up your system?’ 

0.692 

<0.001 

‘How would you rate your comfort level at this point with 
accessing the Internet?’ 

0.571 

0.001 

‘How would you rate your comfort level at this point with 
quiz feature being used to teach this course?’ 

0.537 

0.002 

‘How would you rate your comfort level at this point with 
word processing?’ 

0.506 

0.004 

‘How would you rate your comfort level at this point with 
sending electronic mail? 

0.491 

0.005 

‘How would you rate your comfort level at this point with 
the technical support available to you through your 
Internet service provider (ISP)?’ 

0.489 

0.005 

‘How would you rate your comfort level at this point with 
the links (i.e., additional web sites) feature being used to 
teach this course?’ 

0.464 

0.009 

‘How effective were the rules of eng^ement? That is, to 
what extent did the rules of engagement help you to 
rmderstand what would be involved in taking the Space 
Systems Technologies and Applications on-line course?’ 

0.456 

0.010 

‘How effective was the corarse overview? That is, to what 
extent did the course overview help you to understand 
what you would learn in the Space Systems Technologies 
and Applications on-line course?’ 

0.445 

0.012 

‘How effective was the course syllabus? That is, to what 
extent did the course syllabus help you to understand what 
you would learn in the Space Systems Technologies and 
Applications on-line course?’ 

0.422 

0.018 

‘How would you rate your comfort level at this point with 
the technical support available to you on CD-ROM?’ 

0.410 

0.022 


Table 15. Significant Correlations and Levels of Significance between the Comfort 

Level with Posting to the Forums and Other Variables (Ordered by Level of 

Significance). 


Twenty-five students (80.65%) felt extremely comfortable in sending e-mail, 
which is to be expected in the J^PS environment since every student is expected to check 
e-mail fi'equently. Within the course context, e-mail was functioning well as one of the 
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key ways of communicating. However, few students had problems with sending e-mail: 
two (6.45%) felt ‘extremely imcomfortable’ and one (3.23%) ‘somewhat uncomfortable’. 


Comfort Level with Sending E-mail 
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Figure 14. Distribution of Responses on the Comfort Level with Sending E-mail. 


N = 31 Mean = 4.55 Median = 5 

Minimum = 1 Maximum = 5 Mode = 5 

This variable had ten significant correlations (Table 16). 

The most significant is the correlation between the comfort level with sending e- 
mail and with word-processing. Other correlations, not as high as the first one, are with 
links usage, accessing the Internet, setting up the system, etc. This is the fifth of those 
‘core’ comfort level variables. 
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Question 

.. ...P. 

p value 

‘How would you rate your comfort level at this point with 
word processing?’ 

0.780 

<0.001 

‘How would you rate your comfort level at this point with 
the links (i.e., additional web sites) feature being used to 
teach this course?’ 

0.540 

0.002 

‘How would you rate your comfort level at this point with 
accessing the Internet?’ 

0.529 

0.002 

‘How would you rate your comfort level at this point with 
setting-up your system?’ 

0.526 

0.002 

How would you rate your comfort level at this point with 
posting to the forum? 

0.491 

0.005 

‘How effective were the rules of engagement? That is, to 
what extent did the rules of engagement help you to 
understand what would be involved in taking the Space 
Systems Technologies and Applications on-line course?’ 

0.439 

0.013 

‘How effective was the course syllabus? That is, to what 
extent did the course syllabus help you to imderstand what 
you would learn in the Space Systems Technologies and 
Applications on-line course?’ 

0.434 

0.015 

‘How effective was the course overview? That is, to viiat 
extent did the course overview help you to understand 
what you would learn in the Space Systems Technologies 
and Applications on-line course?’ 

0.415 

0.020 

How would you rate your comfort level at this point with • 
the email feature being used to teach this course? 

0.410 

0.022 

‘How would you rate your comfort level at this point widi 
the forum feature being used to teach this course?’ 

0.392 

0.029 


Table 16. Significant Correlations and Levels of Significance between the Comfort 
Level with Sending E-mail and Other Variables (Ordered by Level of Significance). 


The next question was open-ended. It asked: ‘In what areas of the on-line 
technologies (system set-up, accessing the internet, word-processing, posting to forums) 
do you still need help?’ Twenty students answered ‘None’ and one who answered ‘None, 
just practice.’ I divided the other responses into two groups: ‘Forum postings’ and 
‘Other’: 
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Foram Postings 

• Post or replying to a comment in the Forum is still less than user friendly 

• Posting to Forums 

• The postings are still rough as well as the quizzes 

• Posting to the forum is still hit or miss. In other words, I'm not confident 
that what I submit is going where I think I am sending it. 

• Posting without loosing work 

• lam still getting run-time script errors when accessing postings 

Other 

• System set-up: advanced issues, troubleshooting 

• Very confusing because of all bugs in this system. Thanks for the 
instructor’s summary via email - this helped me stay on track. 

• At this time was changing the subject type, but that has all been resolved.^ 

To conclude, the students’ answers to this question indicated that problems with 
forum postings were most common, but the majority of them did not have significant 
problems with on-line technology that had been not solved when they answered the 
questionnaire. 

The sixth comfort level scale was about software technical support: ‘How would 
you rate your comfort level at this point with the technical support provided for using the 
WebCT software?’ The distribution of the student responses can be seen in Figure 15. 


5 This response is probably also related to forum posting issues. 
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Figure 15. Distribution of Responses on Comfort Level with the Technical Support 

Provided for the WebCT Software, 

N = 31 Mean = 3.61 Median = 4 

Minimum = 2 Maximum = 5 Mode = 4 

A relatively large number of neutral answers (twelve or 38.71%) indicates many 
students did not need to use the technical support for the WebCT software. Those who 
needed that support were mostly ‘somewhat’ or ‘extremely’ comfortable, i.e., satisfied 
with the support provided. Only diree students were ‘somewhat uncomfortable’, i.e., not 
satisfied with the technical support. 

This variable is significantly correlated only with three other variables in the 
instrument (Table 17). 
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Question 

P 

p value 

‘How effective were module objectives? That is, to what 
extent did the module objectives help you to understand 
what you would learn in Ae Space Systems Technologies 
and Applications on-line course?’ 

0.462 

0.009 

‘How effective were the rules of engagement? That is, to 
what extent did the rules of engagement help you to 
understand what would be involved in taking the Space 
Systems Technologies and Applications on-line course?’ 

0.408 

0.023 

‘How would you rate your comfort level at this point with 
the email feature being used to teach this coiuse?’ 

0.406 

0.023 


Table 17. Significant Correlations and Levels of Significance between the Comfort 
Level with the Technical Support Provided for the WebCT Software and Other Variables 

(Ordered by Level of Significance). 

The students more satisfied with the technical software support tended to estimate 
some course features set by the instructor (module objectives and rules of engagement) as 
more effective (i.e., they had more utility and fewer difficulties with these features). The 
third relation shows that lower comfort level with technical software support was 
associated with lower comfort level with e-mail features in the course. 

The tenth variable in the instrument asked the students the following: ‘How would 
you rate your comfort level at this point with the technical support available to you 
through your Internet service provider (ISP)?’ Figure 16 below shows their responses. 
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Figure 16. Distribution of Responses on Comfort Level with the Technical Support 

Provided by the ISP. 

N = 31 Mean = 3.74 Median = 4 

Minimum = 2 Maximum = 5 Mode = 3 

This distribution was very similar to the responses to the WebCT technical 

support question, with a slight difference. There were more neutral answers (fourteen or 

45.16% did not need any help from their ISP) and only one uncomfortable student. This 

variable had significant correlation with three other variables (Table 18). 
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Question 

P 

p value 

‘How would you rate your comfort level at this point with 
posting to the forum?’ 

0.489 

0.005 

‘How effective was the course syllabus? That is, to what 
extent did the course syllabus help you to understand what 
you would learn in the Space Systems Technologies and 
Applications on-line course?’ 

0.399 

0.026 

‘How effective was the course overview? That is, to what 
extent did the course overview help you to understand 
what you would learn in the Space Systems Technologies 
and Applications on-line course?’ 

0.395 

0.028 


Table 18. Significant Correlations and Levels of Significance between the Comfort 
Level with the Technical Support Provided by the ISP and Other Variables (Ordered by 

Level of Significance). 

Found correlations are statistically significant, but mild. Students comfortable 
with ISP support tended to feel more comfortable with posting e-mail and with the course 
features of the syllabus and course overview. 


The following question was about student comfort level with automated data 
processing (ADP) office technical support. The responses were distributed as follows in 
Figure 17. Eighteen responses (58.06%) were neutral. It seems students did not have to 
rely very much on these support tools. Only two students (6.45%) students said they were 
‘somewhat uncomfortable’. 


Only one variable was statistically significantly correlated with this variable. It 
was the comfort level with die course overview feature (‘How effective was the course 
overview? That is, to what extent did the course overview help you to understand what 
you would learn in the Space Systems Technologies and Applications on-line course?’). 
Spearman’s coefficient of rank correlation was p = 0.405, and the significance level p = 
0.024. 
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Figure 17. Distribution of Responses on the Comfort Level with the Technical 

Support Provided by the ADP Office, 

N = 31 Mean = 3.35 Median = 3 

Minimum = 2 Maximum = 5 Mode = 3 

The instructor was also interested in the student comfort level with the ‘Pegasus’ 

CD-ROM provided by CFU. The question was: ‘How would you rate your comfort level 

at this point with the technical support available to you on CD-ROM?’ Figure 18 shows 

how the student responses were distributed. 


Comfort Level with 'Pegasus' CD-ROM 
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Figure 18. Distribution of Responses on the Comfort Level with die Technical 

Support Provided with the CD-ROM. 
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N = 31 Mean = 3.19 Median = 3 

Minimum = 2 Maximum = 5 Mode = 3 

Most students (25 students or 80.65%) responded neutrally to this question. Only 
six responses were non-neutral, and one of these six was negative (‘somewhat 
uncomfortable’). However, four correlations between this and other variables were 
statistically significant (Table 19). 


Question 

P 

msm 

‘How would you rate your comfort level at this point with 
the forum feature being used to teach this course?’ 

0.472 

0.007 

‘How would you rate your comfort level at this point with 
posting to the forum?’ 

0.410 

0.022 

‘How effective were the rules of engagement? That is, to 
what extent did the rules of engagement help you to 
understand what would be involved in taking the Space 
Systems Technologies and Applications on-line course?’ 

0.392 

0.029 

‘How would you rate your comfort level at this point with 
the links (i.e., additional web sites) feature being used to 
teach this course?’ 

0.362 

0.045 


Table 19. Sigmficant Correlations and Levels of Significance between the Comfort 
Level with the CD-ROM and Other Variables (Ordered by Level of Significance). 


The first correlation tells us that same students who had a lower comfort level 
with the CD-ROM tended to have a lower comfort level with forums in general; with 
posting to forums, and with the links feature. Also, they tended to see rules of 
engagement less effectively than other students. 

Next was the open-ended question about other technical problems. Its text was: 
‘What technical issues (e.g., those related to WebCT, Internet Services, Local ADP, CD- 
ROM, etc.) do you still need to resolve?’ Nineteen students responded ‘None’. I grouped 
the rest of responses into three main groups: 
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Quiz 


• The grading problem with the quizzes 

• The only issue is problem with the quizzes. However, I know that's being 
worked 

• It's frustrating that the quiz portion is still not functioning properly - I do 
not want to get too far ahead into the course and then have to back up and 
remediate sevanl quizzes at one time once it get fixed 

I Haven’t Started Yet^ 

• Don’t know yet! 

• Haven't had to use any 

• All answers should be 'n/a' because I have yet to use them 

Miscellaneous 

• The inability to use certain links from the coxirse schedule, and the 
problems with scoring quizzes 

• Some of the questions do not pertain in Qll. ‘N/a’ should be given as a 
response option'^ 

• Just glitches everyone else is having (with WebCT) 

• None other links than those that are group problems. (Not being able to 
take the quizzes and the schedule links.) 

• Problems logging on from AOL 

• WhatCD?8 

After this, there was another open-ended question about any other possible 
thoughts on this aspect of the course (‘Please provide any additional thoughts.’). Twenty- 
seven students did not have any comments and four students did. Here are the comments; 


^ This means that these students didn’t have opportunity to &ce any technical problems yet. 
Comment about this questionnaire. 

* This student didn’t have any information aboxtt ‘Pegasus’ CD-ROM. 
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Negative Comment 


• In general, tech support sucks. Most of these people can't think beyond the 
canned answers for them within a computer thus making the support they 
can provide you minimal and, ultimately, useless. Which is not to say that 
good tech support is not available. Just that it's not available for the 
categories you listed and the companies that provide them (Pacific Bell, 
NPS). 

Neutral Comments 

• The WebCT interface still needs some work 

• The problems that I usually encounter are the same that other are 
encountering 

Positive Comment 

• Help has been great! 


The next eight five-point scale questions asked the students how comfortable they 
were with different features used in the course: forums, e-mail, rules of engagement, etc. 

The first question was about the forum feature: ‘How woxild you rate your 
comfort level at this point with the forum feature being used to teach this course?’ Figure 
19 shows the distribution. 

An overwhelming majority of the students was feeling ‘somewhat comfortable’ 

(17 students or 54.84%) or ‘extremely comfortable’ (seven students or 22.58%) with the 

0 

forums. Only fom* (12.90%) felt imcomfortable (one of them ‘extremely uncomfortable’), 
and three other students were neutral. Interestingly, their responses to fiiis question are 
similar but not identical to the question about the comfort level with posting to forums 
(Figure 13). Briefly, more of them were ‘extremely comfortable’ with posting to the 
forums than with the use of the forums in the course. 
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Figure 19. Distribution of Responses on the Forums Used in the Course. 

N = 31 Mean = 3,84 Median = 4 

Minimu m = 1 Maximum = 5 Mode = 4 

The calculation showed that ten rank correlations between this and other variables 
were statistically significant (Table 20). 

The highest correlation is between two variables asking about the comfort level 
with posting to the forums and about the comfort level with the use of forums in the 
course. Setting-up the systems, accessing the Internet, using word-processing, and 
comfort with sending e-mail appeared to be important issues correlated with the comfort 
of use of the forums and with a few other variables in this set. 
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Question 

P 


‘How would you rate your comfort level at this point with 
posting to the forum?’ 

0.800 

<0.001 

‘How would you rate your comfort level at this point with 
setting-up your system?’ 

0.614 

<0.001 

‘How would you rate your comfort level at this point with 
accessing the Internet?’ 

0.587 

0.001 

‘How would you rate your comfort level at this point with 
word processing?’ 

0.503 


‘How would you rate your comfort level at this point with 
the email feature being used to teach this course?’ 

0.491 

0.005 

‘How would you rate your comfort level at this point with 
the technical support available to you on CD-ROM?’ 

0.472 

0.007 

‘How would you rate your comfort level at this point with 
the links (i.e., additional web sites) feature being used to 
teach this course?’ 

0.406 

0.023 

‘How would you rate your comfort level at this point with 
sending electronic mail?’ 

0.392 

0.029 

‘How would you rate your comfort level at this point with 
quiz feature being used to teach this course?’ 

0.360 

0.047 


Table 20. Significant Correlations and Levels of Significance between the Comfort 
Level with Use of the Forums in the Course and Other Variables (Ordered by Level of 

Significance). 

Next was the question about the comfort level with the use of e-mail features in 
the course (Figure 20). 


Comfort Level with E-mail Feature in the Course 
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Figure 20. Distribution of Responses on the Comfort Level with the E-mail Feature 

in the Course. 
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N = 31 Mean = 3.98 Median = 4 

Minimum = 1 Maximiim = 5 Mode = 5 

There is a very obvious difference between this distribution of responses and the 
responses to the question about the comfort level with sending e-mail (Figure 14), where 
over 26 students (83.87%) responded to being ‘extremely comfortable’. Here only 
thirteen students (41.94%) chose the same response. It is not clear >Aiiy so many students 
(nine or 29.03%) chose a neutral answer. It could be because of the frequency, content, or 
other characteristics of e-mail messages they were receiving from the instructor, or they 
did not quite understand the question. Table 21 shows the correlations between this and 
other variables. Correlations are moderate. 


Question 


p value 

‘How would you rate your comfort level at this point with 
the forum feature being used to teach this covurse?’ 

0.491 

0.005 

‘How effective were module objectives? That is, to what 
extent did the module objectives help you to understand 
viiat you would learn in die Space Systems Technologies • 
and Applications on-line course?’ 

0.491 

0.005 

‘How would you rate yom comfort level at this point with 
accessing the Internet?’ 

0.477 

0.007 

‘How effective was the course overview? That is, to what 
extent did the course overview help you to understand 
what you would learn in the Space Systems Technologies 
and Applications on-line course?’ 

0.436 

0.014 

‘How would you rate your comfort level at this point with 
word processing?’ 

0.425 

0.017 

‘How would you rate your comfort level at this point with 
sending electronic mail?’ 

0.410 

0.022 

‘How would you rate your comfort level at this point with 
the technical support provided for using the WebCT 
software?’ 

0.406 

IQQH 


Table 21. Significant Correlations and Levels of Significance between the Comfort 
Level with the E-mail Feature in the Course and Other Variables (Ordered by Level of 

Significance). 
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As in many other similar cases, several different variables with the comfort level 
answers about various aspects of the course (forums, accessing the Internet, word 
processing) and course features (course overview, module objectives) are correlated with 
medium to lower-medium coefficients^. 

The following question asked students about the quiz feature: ‘How would you 
rate your comfort level at this point with the quiz feature being used to teach this course?’ 
Figure 21 shows the distribution of the responses. 

Interestingly, in the distribution of the student responses students split into two 
different groups: approximately one third of the students (ten students or 32.26%) said 
they felt uncomfortable with the quiz feature (half of them ‘somewhat’, another half 
‘extremely uncomfortable’), and two thirds (nineteen students or 61.29%) comfortable. 
Only two students chose a neutral response. The fact that during the course there were 
some technical problems with the quizzes only partially explains such a distribution. 
Also, the instructor’s notes suggest that the students had different interpretations of the 
importance of the quizzes in determining their final grade. This phenomenon will be 
examined in the section on interviews with students. 

This variable has statistically significant rank correlations with six other variables 
in the instrument (Table 22). 


^ Statisticians usually consider correlations over 0.75 or 0.80 as ‘high’ and tiiose below 0.50 or 0.45 as 
‘low’. They do not take into consideration at all any correlation if it is not statistically significant with at 
least a previously determined level of significance. 
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Comfort Level with Quiz Feature in the Course 
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Figure 21. Distribution of Responses on the Comfort Level with the Quiz Feature in 

the Comse. 

N = 31 Mean = 3.45 Median = 4 

Minimum = 1 Maximum = 5 Mode = 5 


Question 

P 

p value 

‘How would you rate your comfort level at this point with 
posting to the forum?’ 

0.537 

0.002 

‘How effective were the rules of engagement? That is, to 
what extent did the rules of engagement help you to 
understand what would be involved in taking the Space 
Systems Technologies and Applications on-line course?’ 

0.531 

0.002 

‘How would you rate your comfort level at this point with 
the links (i.e., additional web sites) feature being used to 
teach this course?’ 

0.401 

0.026 

‘How would you rate your comfort level at this point with 
accessing the Internet?’ 

0.399 

0.026 

‘How would you rate your comfort level at this point with 
word processing?’ 

0.388 

0.031 

‘How woxild you rate your comfort level at this point with 
the forum feature being used to teach this course?’ 

0.360 

0.047 


Table 22. SignilBcant Correlations and Levels of Significance between the Comfort 
Level with the Qixiz Feature in the Course and Other Variables (Ordered by Level of 

Significance). 
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The next question was: ‘How would you rate your comfort level at this point with 
the links (i.e., additional web sites) feature being used to teach this course?’ The 
distribution of the responses is shown in Figure 22. 

Comfort Level with Links Feature in the Course 
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Figure 22. Distribution of Responses on the Comfort Level with the Links Feature in 

the Course. 

N = 31 Mean = 4.45 Median = 5 

Minimum = 1 Maximum = 5 Mode = 5 

The vast majority of the students (twenty-one students or 67.74%) were 
‘extremely comfortable’ with links as an important component of the on-line course 
material. Less (seven or 22.58%) were ‘somewhat comfortable’, and only two of them 
indicated they felt uncomfortable. Of all the course features we analyzed so far in this 
chapter, links clearly had the highest comfort level among the students. 

Table 23 lists eleven variables that have statistically significant rank correlations 
with the links-related comfort level. The comfort level with the links feature highly 
correlates with the comfort level in accessing the Internet, and the correlations are 
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somewhat lower with the comfort level with word processing, sending e-mail, and other 
on-line course features like rules of engagement and module objectives. 

Another element of on-line course materials was the comse overview. The 
question was: ‘How effective was die course overview? That is, to what extent did the 
course overview help you to understand what you would learn in the Space Systems 
Technologies and Applications on-line course?’ Figure 23 shows the distribution of the 
responses. 


Question 

P 

p value 

‘How would you rate your comfort level at this point with 
accessing the Internet?’ 

0.728 

<0.001 

‘How effective were the mles of engagement? That is, to 
what extent did the rules of engagement help you to 
understand what would be involved in taking the Space 
Systems Technologies and Applications on-line course?’ 

0.629 

<0.001 

‘How would you rate your comfort level at this point with 
word processing?’ 

0.541 

0.002 

‘How would you rate your comfort level at this point with 
sending electronic mail?’ 

0.540 

0.002 

‘How effective were module objectives? That is, to what 
extent did the module objectives help you to understand what 
you would leam in the Space Systems Technologies and 
Applications on-line course?’ 

0.524 

0.002 

‘How effective was the course syllabus? That is, to what 
extent did the course syllabus help you to understand what 
you would leam in the Space Systems Technologies and 
Applications on-line course?’ 

0.529 

0.003 


0.499 

0.004 

‘How would you rate your comfort level at this point widi 
posting to the forum?’ 

0.464 

0.009 

‘How would you rate your comfort level at this point with 
the forum feature being used to teach this course?’ 

0.406 

0.023 

‘How would you rate your comfort level at this point with 
quiz feature being used to teach this course?’ 

0.401 

0.026 

‘How would you rate your comfort level at this point witii 
the technical support available to you on CD-ROM?’ 

0.362 

0.045 


Table 23. Significant Correlations and Levels of Significance between the Comfort 

Level with the Link Feature in the Course and Other Variables (Ordered by Level of 

Significance). 
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Figure 23. Distribution of Responses on the Comfort Level with the Course 

Overview. 


N = 31 Mean = 4.16 Median = 4 

Minimum = 2 Maximum = 5 Mode = 4 

Only three students (9.67%) felt ‘somewhat uncomfortable’ with the course 
overview, and three students (9.67%) chose neutral statements. Most of the students (14 
or 45.16%) felt ‘somewhat comfortable’, and the remaining students (12 or 38.71%) even 
‘extremely comfortable’, which gives very encouraging feedback to the course instructor. 
The correlations between this and other variables are shown in Table 24. 

These correlations clearly show a significant level of association between all 
additional on-line course content created by the instructor to facilitate student learning: 
course overview, rules of engagement, module objectives, and course syllabus. In other 
words, the same students who felt imcomfortable with any of these textual features used 
in the course, tended to have a similar impression about some of the other content areas 
in the course. Other variables, those reflecting ‘core on-line issues’, such as the comfort 
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level with accessing the Internet or with word processing, and some others such as the 
comfort level with ADP office support, were significant. 


Question 

P 

p value 

‘How effective were the rules of engagement? That is, to 
what extent did the rules of engagement help you (...)?’ 

0.621 

<0.001 

‘How would you rate your comfort level at this point with 
accessing the Internet?’ 

0.581 

0.001 

‘How effective was the course syllabus? That is, to what 
extent did die course syllabus help you (...)?’ 

0.569 

0.001 

‘How effective were module objectives? That is, to what 
extent did the module objectives help you (...)?’ 

0.515 

0.003 

‘How would you rate your comfort level at this point with 
the links (i.e., additional web sites) feature being used to 
teach this course?’ 

0.499 

0.004 

‘How would you rate your comfort level at this point with 
posting to the forum?’ 

0.445 

0.012 

‘How would you rate your comfort level at this point with 
the email feature being used to teach this course?’ 

0.436 

0.014 

‘How would you rate your comfort level at this point with 
word processing?’ 

0.424 

0.017 

‘How would you rate your comfort level at this point with 
setting-up your system?’ 

0.421 

0.018 

‘How would you rate your comfort level at this point with 
sending electronic mail?’ 

0.415 

0.020 

‘How would you rate your comfort level at this point with 
the technical support available to you through the school 
ADP office?’ 

0.405 

0.024 

‘How would you rate your comfort level at this point with 
the technical support available to you throu^ your 
Internet service provider (ISP)?’ 

0.395 

0.028 


Table 24. Significant Correlations and Levels of Significance between the Comfort 
Level with the Course Overview and Other Variables (Ordered by Level of Significance). 

The next item was: ‘How effective were module objectives? That is, to what 
extent did the module objectives help you to understand what you would learn in the 
Space Systems Technologies and Applications on-line course?’ The distribution of the 
student responses is shown in Figure 24. 
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Comfort Level with Module Objectives 



N = 31 Mean = 4.06 Median = 4 

Minimum = 2 Maximum = 5 Mode = 4 

This distribution is very similar to the distribution of the responses on the comfort 
level with the course overview (Figure 23), except there were somewhat more neutral 
responses. Only one student felt somewhat imcomfortable with module objectives. 

Eight correlations were significant. Similar comments apply to this question about 
comfort level with module objectives as with the previous one: correlations between 
additional on-line course contents (‘course overview’ and ‘course syllabus) and ‘core on¬ 
line issues’ dominate (Table 25). A similar tendency can be expected with responses to 
the next two questions also. 
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Question 

P 

P value 

‘How effective were the rules of engagement? That is, to 
what extent did the rules of engagement help you (...)?’ 

0.565 

0.001 

‘How would you rate your comfort level at this point with 
the links (i.e., additional web sites) feature being used to 
teach this course?’ 

0.524 

0.002 

‘How effective was the course syllabus? That is, to what 
extent did the course syllabus help you (...)?’ 

0.523 

0.003 

‘How effective was the course overview? That is, to what 
extent did the course overview help you (...)?’ 

0.515 

0.003 

‘How would you rate your comfort level at this point with 
the email feature being used to teach this course?’ 

0.491 

0.005 

‘How would you rate your comfort level at this point with 
the forum feature being used to teach this course?’ 

0.473 

0.007 

‘How would you rate your comfort level at this point with 
the technical support provided for using the WebCT 
software?’ 

0.462 

0.009 

‘How would you rate your comfort level at this point with 
accessing the Internet?’ 

0.406 

0.023 


Table 25. Significant Correlations and Levels of Significance between the Comfort 
Level with the Module Objectives and Other Variables (Ordered by Level of 

Significance). 

The next question was: “How effective were the rules of engagement? That is, to 
what extent did the rules of engagement help you to understand what would be involved 
in taking the ‘Space Systems - Technologies and Applications’ on-line course?” The 
distribution of the student responses is in Figure 25. 
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Figiire 25. Distribution of Responses on the Comfort Level with the Rules of 

Engagement. 


N —31 Mean = 4.19 Median = 4 

Minimum = 2 Maximum = 5 Mode = 4 

This distribution follows a similar pattern as the previous two. There were more 
‘somewhat comfortable’ responses: 51.61% to this question compared to the previous 
two questions (45.16%). Only one student expressed feeling uncomfortable with the rules 
of engagement in the course. 

Table 26 shows all significant correlations between this and the other variables. 


80 





Question 

P 

p value 

‘How effective was the course syllabus? That is, to what 
extent did the course syllabus help you (...)?’ 

0.718 

<0.001 

‘How would you rate your comfort level at this point with 
the links (i.e., additional web sites) feature being used to 
teach this course?’ 

0.629 

<0.001 


0.621 

<0.001 

‘How effective were module objectives? That is, to what 
extent did the module objectives help you (...)?’ 

0.565 

0.001 

‘How would you rate your comfort level at this point with 
quiz feature being used to teach this course?’ 

0.531 

0.002 

‘How would you rate your comfort level at this point with 
posting to the forum?’ 

0.456 

0.010 

‘How would you rate your comfort level at this point with 
accessing the Internet?’ 

0.451 

0.011 

‘How would you rate your comfort level at this point with 
word processing?’ 

0.450 

0.011 

‘How would you rate your comfort level at this point with 
setting-up your system?’ 

0.441 

0.013 

‘How would you rate your comfort level at this point with 
sending electronic mail?’ 

0.439 

0.013 

‘How would you rate your comfort level at this point with 
the technical support provided for using the WebCT 
software?’ 

0.408 

0.023 

‘How would you rate your comfort level at this point with 
the technical support available to you on CD-ROM?’ 

. 0.392 

0.029 


Table 26. Significant Correlations and Levels of Significance between the Comfort 
Level with the Rules of Engagement and Other Variables (Ordered by Level of 

Significance). 


Correlations followed a very similar pattern as discussed above. 

The last five-point scale question in this instrument was: “How effective was the 
course syllabus? That is, to what extent did the course syllabus help yOu to xmderstand 
what you would learn in the ‘Space Systems Technologies and Applications’ on-line 
course?” Figure 26 shows the distribution of the responses. 
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Comfort Level with Course Syllabus 



Figure 26. Distribution of Responses on the Comfort Level with the Coiuse Syllabus. 

N = 31 Mean = 3.84 Median = 4 

Minimum = 1 Maximum = 5 Mode = 4 

Three students (9.68%) felt ‘uncomfortable’ about the course syllabus, and seven 
(22.58%) chose a neutral response. The latter students probably did not carefully examine 
the syllabus. Twenty-two (70.97%) felt ‘comfortable’ (almost equally divided between 
‘somewhat’ and ‘extremely comfortable’. Correlations formed a similar pattern for the 
comfort level with the course overview, module objectives, and rules of engagement. 
Table 27 shows coefficients and significance levels. 

These correlations also followed the expected pattern. 

At the end of the questionnaire, there were five open-ended questions asking the 
students about needed help with the web-based features, general liking of the course, 
concerns, suggestions, and question on how they would create their own on-line course if 
they were instructors. 
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Question 

P 

p value 

‘How effective were the rules of engagement? That is, to 
'wiiat extent did the rules of engagement help you (...)?’ 

0.718 

<0.001 

‘How effective was the course overview? That is, to what 
extent did the course overview help you (...)?’ 

0.569 

0.001 

‘How effective were module objectives? That is, to vdiat 
extent did the module objectives help you (...)?’ 

0.523 

0.003 

‘How would you rate your comfort level at this point with 
the links (i.e., additional web sites) feature being used to 
teach this course?’ 

0.509 

0.003 

‘How would you rate your comfort level at this point with 
sending electronic mail?’ 

0.434 

0.015 

‘How would you rate your comfort level at this point with 
posting to the forum?’ 

0.422 

0.018 

‘How would you rate your comfort level at this point with 
setting-up your system?’ 

0.401 

0.026 

‘How would you rate your comfort level at this point with 
the technical support available to you through your 
Internet service provider (ISP)?’ 

0.399 

0.026 

‘How would you rate your comfort level at this point with 
accessing the Internet?’ 

0.398 

0.027 


Table 27. Significant Correlations and Levels of Significance between the Comfort 
Level with the Course Syllabus and Other Variables (Ordered by Level of Significance). 

The first of these open-ended questions was the question about help the students 
might need: ‘In what areas do you still need help with the web-based technological 
features (forum, email, quizzes, links, etc) used to teach this on-line course?’ Twenty 
students answered they needed no such help. Here are the other answers, systematized: 

Quiz 

• None - just fix the quiz section 

• Fix quiz 2, get quiz 3 up and running 

• I await resolution of the quiz technical difficxilties 

• The quizzes obviously need to be repaired technically 

• Quizzes - we just want die site fixed 

• Quizzes need a summary which questions are wrong and unanswered 
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Other and unclear 

• Forum posting 

• There are still some minor bugs 

Neutral 

• I haven’t used e-mail. Quiz grading is much better 

The responses clearly suggest that at the moment of responding to this on-line 
questionnaire, the only significant technical problem that the students needed help with 
was the ‘bug’ in quiz scoring (six of the ten responses addressed this issue). 

The second question asked students if they liked the course: ‘So far, what do you 
like about this on-line course?’ Here is how the students responded: 

Time management flexibility and accessibility^*^ 

• Accessibility 

• Can work on it at my leisure 

• Convenience to do it on my schedule 

• Ease of access... Able to access aroimd my schedule 

• Flexibility of my schedule - ability to review - additional web links to 
areas of interest 

• Flexibility! [Three answers] 

• Freedom to work at own pace 

• I can do it anytime 

• I like the pace. However, I'm not sure if I'm learning more or less than 
what is expected. 

• Self paced with visible objectives and requirements 

• Self-paced. You can work it into your schedule easier 


These two types of answers were combined together because, according to student responses, they 
seem to be very close to each other widiin the context of die on-line course. 
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• The ability to manage your own time 

• The convenience of managing my time as it relates to the requirements of 
the class. I can spend extra time or less time is required. 

• The convenience of the course, i.e. being able to do the course whenever I 
choose 

• The flexibility to learn when I am ready to 

• The freedom of time... 

• The autonomy. I like doing the work on my own so I can skim what I 
already know and go deeper on things that interest me. 

Forums 

• The forum interaction is great, but I would agree with the posting that we 
should meet once a week or so at the later stages of the course to have 
question/answer periods with the instructor 


Multiple answers and miscellaneous 

• I can work on it all hours and at my home PC. Also the continuous 
postings to the forums are beneficial. 

• I like the forums format. I also like the links in the modules 

• I like the layout and interface real well. Although the text is extremely 
extensive, it has been informative and enjoyable to read so far. 

• I like the links and forum feature 

• I like the links to sites that I may not have found otherwise. The book 
seemed a tad bit too simple at first, but as I got to the orbit section, there is 
some in-deplh material. 

• Multiple-choice open book repeatable quizzes 

• Still working 

• The ability to crawl out of the rack and check the forums, and make 

comments. The ability to do the work to better fit my schedule. 

• The text and the links 

• Very insightful discussions in the forum; excellent support for technical 
issues; a very responsive instructor; and links to very interesting web sites 


Clearly, the students outlined time flexibility and accessibility of the course 

material whenever it suits them as main reason for liking the course (19 out of 30 
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responses or 63.33% of the responses) The same issues are also mentioned in three of the 
answers in the third group (Multiple Answers and Miscellaneous). 


The next question asked students about concerns they had about the course: ‘So 
far, what are your concerns about this on-line course?’ Eleven students responded ‘None’ 
or did not respond at all which implies they had no significant concerns. Other responses 
are systematized as follows: 


Forum 

• Too many forum postings! It’s just too difficult and time consuming to 
realistically read everything that everyone posts. Forum groups need to be 
broken down (say, 5-7 people) so that we can share ideas in a smaller 
group. Also, the fact that we are getting behind with Modules that don't 
work is frustrating. It’s as if we’re at a standstill and will need to catch up 
later on. 

Comparison with face-to-face class 

• Depthofknowledge vs. in-class instruction 

• I am concerned that I may not be getting the same information as my 
counterparts in the in-room class 

• See above #25. I’ve learned some things... Not sure if I’m getting the 
same value as students who are actually sitting in a classroom with a 
professor. 

Learning pace and organization 

• Getting too far ahead in the assignments without being able to keep up 
with the quizzes because that section is not working properly 

• Keeping up on the readings and not falling behind 

• My personal lack of urgency for the first couple of weeks. And my 
imwillingness to read something posted over a paragraph or two in length. 

Technical problems 

• Run-time errors... They are not due to my memory allocation because I'm 
getting them no matter what computer I’m accessing the web site fi-om... 
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• The apparent slowness with which the WebCT has been able to resolve the 
technical problems associated with the quizzes 

• The password and user id issue, but that has been addressed 

• Too many problems up front that should have been worked out ahead of 
time 

Grading 

• How will we be graded? 

• I'm concerned with how we will be evaluated given that we're having 
problems with scoring the quizzes. 

• That some wordy students will get better grades, even when the context is 
not worthy. I hope that die grade is based on the quality of the content, not 
the quantity. A lot of the comments are verbose and boring. 

Multiple answers and miscellaneous 

• My only concern so far is about missing an event. Since some of the items 
are not working, I may overlook a part this is working. 

• The lack of interaction with the professor 

• Too cumbersome to always check for replies... Also sometimes ready to 
take quiz or evaluation and it has not been posted yet. 

Positive opinions 

• No negative concerns, I am very positive and motivated to continue into 
deep space! 

• That there will be no more after this. It is a great medium for learning 

There was no one predominant issue students addressed, but students expressed 
concerns about different themes. Some responses seem to be unrelated to the specific DL 
nature of the course, but were more general student concerns (learning pace and 
organization) that appear in any type of class. Student concerns are grouped around the 
following issues: technical problems (four comments); uncertainty whether the 
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possibilities of DL are equal to those is face-to-face class (three); grading (three); forums 
(one) and mixed/miscellaneous problems (three comments). 

The next question put students into the position of thinking how they would 
organize a DL course: ‘Please respond to the following question. If I were building an on¬ 
line course to teach space systems-related instruction, I would...’ The students gave 
many responses to this question, and after analysis of their content I formed six different 
groups of responses based on similar themes in their comments. Here are the grouped 
responses: 

Just like this course 

• ... Do exactly what you are doing. 

• Copy that format! 

• Do exactly what you're doing. I think we're getting a good overview of 
Space and are taking away a lot of useful information. 

• Model it pretty much like you have 

• Not change a damn thing 

• Try to do as good as you have... Really! It is a good program with a 
layout that makes navigation easy. 

I’m too new to say 

• Don’t know 

• lam just starting; ask me at the end 

• I haven’t really thought about it, but I am a novice at on-line courses, so 
my input wouldn't really contribute anything substantial at this time 

Hire someone 

• Find experts on distance learning through technology (which has been 
done here already) 

• I would hire someone to do it, because it's beyond me 
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First, fix what you can 


Conduct thorougji system testing and evaluation and keep the class small 
till all the bugs are worked out 

Ensure diat the various pieces were working before beginning. This is not 
meant as sarcastically as it sounds. 

Ensure the system is up and running before starting a class with students. I 
would require a ‘dry run’ of the entire course and all features completed 

rd get a better feel for how it is going to work before putting online. 
Possibly hold this 'online' class IN class, using it as a group first. Seems to 
be too many unknowns. Also, it would REALLY benefit us to be able to 
meet firom time to time to assess the class and where all of us stand. I 
understand that the point of the class is so the distance learning can occur, 
but since the class is in its genesis, perhaps meeting once every two weeks 
would provide insight that the forum cannot. 

Keep fixing the bugs and add more user help key like spell check and 
summary charts for graded answers 

Work out the bugs ahead of time -1 think you have done a great job with 
the layout and interface thus far 

Content improvements 

Add CCTV/tape like instruction by instructor on key points 

Arrange online chats with industry specialists. Follow the same format as 
presented (but without the bugs). 

Have more mathematical or practical applications 

Provide links (outsource or internal) that would provide a hands-on 
approach to solving some of the questions/problems, i.e., orbital model 
that actually used the information that we input 


Instruction methods improvements 

Continue to require the Professor to keep students on track via email. 
Those emails add confidence and understanding to the student as what the 
instructor exactly expected them to accomplish. A syllabus is nice, but as 
always changes occur. In an online course, little changes create confusion 
even worse that in regular class. Those emails firom the professor help. 

Ensure that complete communication via email is always available letting 
me know when forums have been replied to, when quizzes and evaluation 
are opened 


• Have to leam a lot more about both space systems and distance education 

• Keep the learning process as well as the testing setup simple and 
straightforward 

• Limit the length of the postings and smaller group forums sooner 

• One thought. In the forum people write very LARGE paragraphs. It is 
difficult to read online and follow. Part of the rules of the road should be 
to write 4/5 sentences and then make a break and continue. Just like I am 
doing now. I would argue people would be more inclined to read 5/6 
blocks of text vise one humungous block of text. 

• Use a great textbook like the one we have and encourage students with 
regular e-mails (good job!) 


The last question in this questionnaire asked students if they had suggestions: ‘Do 
you have any suggestions, at this point (in particular, about your orientation to taking a 
course on-line)?’ Eighteen of them did not respond or answered ‘None’, ‘Nothing yet’, or 
‘Nothing at this time’. Three students responded that they have answered this question 
previously. Here are the answers of the remaining students: 

Advice and ideas 

• It seems like the bases are pretty well covered. We can interact online and 
get questions answered quickly. 

• Just some more QC* ^ on the links and quizzes 

• Limit forum size to 10 people (Forum teams A, B, C, D... Gold, Blue, 
Red, Green... Whatever... Poring through a plethora of responses can be 
somewhat tiring... 

• The forum discussions were lagging during the technical chapters 

Positive opinions 

• A hesitant ‘I’d do it again based on this course’. The caveat is that some 
courses (like Calculus) wouldn't lend themselves as well to online 
teaching (in my humble opinion). 

• Make sure requirements of the students are clearly defined at all times 


Quality control. 
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• The orientation was good and the tech support has been very responsive 

Neutral 

• None at this time. I think it is a good idea to combine the quizzes when 
technical difficulties are encountered; it keeps things tracking along to the 
next module. 

Second questionnaire - Summary 

Statements on comfort levels showed that positive responses dominate, but the 
distributions of the student responses are not the same in all cases. For instance, the 
highest level of comfort the students had was with accessing the Internet. Issues like 
setting up one’s own system, posting to forums, or WebCT technical support had some 
negative responses also. Some statements clearly had the highest level of comfort, but 
some students gave negative responses anyway as in the statements about using word 
processing or sending e-mail. 

A few statements produced an exceptionally large number of neutral responses 
such as the statement on ISP technical support or on ADP office support. This occurred 
because most students did not use those services. 

The open-ended question about needed technical help clearly indicated that 
posting to forums was a central problem. 

Eight statements asked the students about the comfort level with the use of 
different features in the course: forums, e-mail, quiz, links, course overview, module 
objectives, rules of engagement, and course syllabus. Distributions differ among these 
variables, although students mostly chose positive responses. Sending e-mail had the 
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most responses of ‘extremely comfortable’, but there were some students who felt 
imcomfortable even with that. 

Asked about additional help with web-based features, six students emphasized 
problems with the quiz, but there were other responses also. 

Another open-ended question asked the students what they liked about the on- lin e 
course. Nineteen comments could be classified as ‘time management flexibility and 
accessibility’. Other features were mentioned also: forums (six times, alone or in 
combinations), links (four responses), quizzes, and support. 

I divided the responses to questions about student concerns into the following 
groups: technical problems (four responses); uncertainty about comparability of DL to 
face-to-face classes (three); learning pace and organization (three); grading (three); 
combinations and other (four). Only two students gave very positive responses to this 
question. 

Near the end of the questionnaire, students were asked to respond to the following 
statement: ‘If I were building an on-line course to teach space systems-related instruction, 
I would...’ Their responses were grouped around: ‘Instructional methods improvements’ 
(seven responses); ‘I would do just same as in this course (six responses); ‘First, fix all 
what you can’ (six); ‘I’m too new to say’ (three); and ‘Hire someone’ (two responses’. 

3. Module 14 Questionnaire 

This questionnaire (Appendix C) was administered immediately after module 14 
was completed, i.e., close to the end of the course (the course had 15 modules). Since 
module 14 was different fi'om the others (special guest and a subject matter expert was 
available for interactions with the students), the instructor wanted to learn how much the 
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students benefited fi:om the module 14 learning objectives. The instructor formulated 
these statements. 

I added several questions about the current comfort level with important features 
used in the course so that the changes in comfort levels fi:om the beginning of the course 
to date could be examined. 

Since the students for this on-line questionnaire were identifiable by name, it was 
possible to compare their responses to this questionnaire widi data jfrom other 
instruments: the first questionnaire, student tracking data, and even final course grades. 

First the students were asked to respond on seven-point scales to seven statements 
about achieving learning objectives in Module 14. The next six statements provided were 
about the student comfort level with features used in the course. At the end of this 
questionnaire, the students were given an opportunity to add their own comments. 

Twenty-six students completed this questionnaire. 

The first question was: ‘Throughout Module 14 you developed and practiced your 
skills in the process of analyzing, articulating, and evaluating DoD Space Control issues. 
The statements below reflect the learning objectives for Module 14. To what extent do 
you agree that the material presented in the Module enabled you to achieve the learning 
objectives? Identifying key elements of Space Control.’ Figure 27 shows the distribution 
of the responses. 
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"identifying Key Elements of Space Control" 



Figure 27. Distribution of the Responses to the First Statement. 

N = 26 Mean = 5.92 Median = 6 

Minimum = 2 Maximum = 7 Mode = 6 


Nineteen students (73.08%) responded with ‘agree’, so this learning objective was 
very successfully met with the module. This variable has significant correlations with 
fourteen variables fi'om this and other instruments (Table 28). 


Instrument 

Statement / Variable 

P 


Module 14 

4 - ‘Participating in forums (e.g. discussing, 
identifying/consulting subject matter experts), to 
evaluate space control issues.’ 

0.559 


Module 14 

9 - ‘How would you rate your comfort level at this 
point with sending electronic mail?’ 

0.494 


Module 14 

8 - ‘How would you rate your comfort level at this 
point with accessing the Internet?’ 

0.470 

0.015 

Module 14 

11 - ‘How would you rate your comfort level at this 
point with quiz feature being used to teach this 
course?’ 

0.440 

0.024 

Fist 

questionnaire 

‘I expect this online course will be valuable and 
improve my learning’ 

0.405 

0.026 

Student 

tracking 

Number of follow-up posts 

0.367 

0.046 


Table 28. Significant Correlations and Levels of Significance between the First 
Statement and other Variables (Ordered by Level of Significance). 
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Correlations in Table 28 show that the students with a higher comfort level in sending e- 
mail, accessing the Internet, and the quiz feature, said that the course material helped 
them in higher achievement in the first learning objective. Also, they tended to have 
somewhat higher expectations to improve their learning throughout the course and to 
send more follow-up posts in forums. 

The second question was: ‘Synthesizing other’s findings in Forums’ (see 
explanatibn with first question). Figure 28 shows the distribution of the responses. 



Figure 28. Distribution of the Responses to the Second Statement. 

N = 26 Mean = 5.19 Median = 5 

Minimum = 5 Maximum = 7 Mode = 5 


Predominant responses to the second statement were ‘slightly agree’ and ‘agree’ 

(ten students or 38.46% chose each). Responses did not have any significant correlations 

with other variables, which would suggest that this learning objective is specific, without 

any apparent relation to any other of the variables measured with this set of instruments. 
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The third statement in the questionnaire was; ‘Interacting with a subject-matter 
expert whose knowledge you can use to solve space-system problems.’ Figure 24 shows 
the distribution of the student responses. 


"Interacting with a Subject Matter Expert.." 



Figure 29. Distribution of the Responses to the Third Statement. 

N = 26 Mean = 4.85 Median = 5 

Minimum = 1 Maximum = 7 Mode = 4 

Here the student responses vary more than the responses to the first two 

statements. Seven students (26.93%) chose neutral answers, which might be an indicator 

that they personally have not taken the opportunity to enter into interactions with guest - 

subject matter expert, or they saw this module in terms of the stated learning objectives 

exactly ‘between successful and unsuccessful’. This variable had significant correlations 

with four other variables (Table 29). 

Students with higher expectations in improving their learning in this course saw 

this interaction as more useful for their learning. Also, they tended to value participation 
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in forums more, and they also tended to participate more in forums by posting messages 
and follow-up posts to forums. 


Instrument 

Statement / Variable 

P 

p value 

Fist 

questionnaire 

‘I expect this online course will be valuable and 
improve my learning’ 

0.450 

0.012 

Module 14 

4 - ‘Participating in forums (e.g. discussing, 
identifying/consulting subject matter experts), to 
evaluate space control issues.’ 

0.439 

0.025 

Student 

tracking 

Number of posted messages 

0.404 

0.027 

Student 

tracking 

Niunber of follow-up posts 

0.414 

0.023 


Table 29. Significant Correlations and Levels of Significance between the Third 
Statement and other Variables (Ordered by Level of Significance). 

The fourth statement was the following: ‘Participating in forums (e.g. discussing, 
identifying/consulting subject matter experts), to evaluate space control issues.’ Figure 30 
shows how the student responses were distributed. Eighteen students agreed (69.23%) - 
six ‘somewhat agreed’ (23.08%), nine ‘agreed’ (34.61%), and three ‘extremely agreed’ 
(11.54%), but there were five neutral (19.23%) and three responses (11.54%) that 
disagreed with the statement. 

Table 30 shows four significant correlations this variable had with other variables. 
Insight into the variables in the table reconfirms the relationship between valuing 
interactivity, valuiug student participation (in forums), and higher expectations in 
learning throughout the couree. A negative sign of correlation between the responses to 
the fourth statement and the question fi-om the first questioimaire about concerns of one’s 
own abilities means that the students less concerned about their own abilities to use web- 
based materials in the on-line class tend to see material in module 14 as helping them 
more in participating in forums and evaluating space control issues. 
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"Participating in Forums, to Evaluate Space Control 

issues" 


9 

8 

7 

6 

Number of s 

students 4 
3 
2 
1 
0 



^e® (.<;e® ^e® -ne® ^^e® ^^e® 


Figure 30. Distribution of the Responses to the Fourth Statement. 


N = 26 Mean = 5.12 Median = 5 

Minimum = 2 Maximum = 7 Mode = 6 


Instrument 

Statement / Variable 

P 

p value 

Module 14 

1 - ‘Identifying key elements of Space Control.’ 

0.559 

■gtliVI 

Module 14 

3 - ‘Interacting with a subject-matter expert whose 
knowledge you can use to solve space-system 
problems.’ 

0.439 

0.025 

Fist 

questionnaire 

‘I expect this online course will be valuable and 
improve my learning’ 

0.392 

0.032 

First 

questionnaire 

‘I am concerned about my ability to use the web- 
based materials in this class’ 

-0.364 

0.048 


Table 30. Significant Correlations and Levels of Significance between the Fourth 
Statement and other Variables (Ordered by Level of Significance). 

All the students agreed with the fifth statement, except two who gave neutral 
responses. Materials in module 14 successfully helped them in learning this learning 
objective (Figure 31). 
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"Articulating How the Concepts of Space Control 
Relate to Joint Vision 2020" 



Figure 31. Distribution of the Responses to the Fifth Statement. 

N = 26 Mean = 5.85 Median = 6 

Minimum = 4 Maximum = 7 Mode = 6 

Four correlations were significant, two on other learning objectives in this 
module, and two on the comfort level with accessing the Internet and with sending e-mail 
(Table 31). Interpretation is not easy, but students with higher comfort levels with basic 
technical tools in the DL course should be expected to be more successful in this kind of 
learning environment. 

The sixth statement was ‘Evaluating the likelihood that various types of space 
countermeasures might be used across file peace-war continuum’. Responses were 
favorable, without ‘disagree’ responses except only one student. Figure 32 shows the 
distribution of the responses. Similarly to other statements in this set, there are few 
significant correlations - two with variables from this same set of statements, one with 
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variable from the first questionnaire, and one with the number of visits to the coiuse 
homepage (Table 32). 


Instrument 

Statement / Variable 

P 

p value 

Module 14 

6 - ‘Evaluating the likelihood that various types of 
space countermeasures might be used across the 
peace-war continuum.’ 

0.525 


Module 14 

7 - ‘Identifying elements of space systems that are 
vulnerable to disruption, degradation or destruction 
by adversaries.’ 

0.448 

0.022 

Module 14 

8 - ‘How would you rate your comfort level at this 
point wiA accessing the Internet?’ 

0.416 

0.035 

Module 14 

9 - ‘How would you rate your comfort level at this 
point with sending electronic mail?’ 

0.388 

0.050 


Table 31. Significant Correlations and Levels of Significance between the Fifth 
Statement and Other Variables (Ordered by Level of Significance). 


"Evaluating tiie Likelihood that Space 
Countermeasures Might be Used" 



Figure 32. Distribution of the Responses to the Sixth Statement. 


N = 26 


Mean = 5.96 Median = 6 


Minimum = 4 


Maximum = 7 Mode = 6 
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Instrument 

Statement / Variable 

P 

p value 

Module 14 

7 - ‘Identifying elements of space systems that are 
vulnerable to disruption, degradation or destruction 
by adversaries.’ 

0.579 

0.002 

Module 14 

5 - ‘Articulating how the concept of space control 
relates to Joint Vision 2020.’ 



Student 

tracking 

Number of visits to course homepage 

0.408 

0.025 

Fist 

questionnaire 

‘How would you rate your on-line search resources 
and techniques?’ 

0.367 

0.046 


Table 32. Significant Correlations and Levels of Significance between Sixth 
Statement and Other Variables (Ordered by Level of Significance). 

The last statement in this set was: ‘Identifying elements of space systems that are 
vulnerable to disruption, degradation or destruction by adversaries.’ Figure 33 shows the 
distribution of the responses, which are in this case also very favorable with only one 
student disagreeing slightly. Significant are the correlations with two other statements 
fi-om this set: the self-perceived on-line search skills firom first questioimaire and the 
comfort level vrith the links feature (Table 33). 


"IdentHying Elements of Space Systems Vulnerabile 
to Disruption, Degradation, or Destruction..." 
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Figure 33. Distribution of the Responses to the Seventh Statement. 
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N = 26 
Minimum = 3 


Mean = 6.23 
Maximum = 7 


Median = 6 
Mode = 6 


Instrument 

Statement / Variable 

.P 


Module 14 

6 - ‘Evaluating the likelihood that various types of 
space countermeasures might be used across the 
peace-war continuum.’ 

0.579 


Module 14 

5 - ‘Articulating how the concept of space control 
relates to Joint Vision 2020,’ 


0.022 

Fist 

questionnaire 

‘How would you rate your on-line search resources 
and techniques?’ 

0.398 

0.030 

Module 14 

12- ‘How would you rate your comfort level at this 
point with the links (i.e., additional web sites) 
feature being used to teach this course?’ 

0.398 

0.044 


Table 33. Significant Correlations and Levels of Significance between the Seventh 
Statement and Other Variables (Ordered by Level of Significance). 

The first set of statements was followed by six questions on the comfort level with 
different features used in the course. These questions are similar or identical to the 
question used in Questionnaire A-2 so that comparisons are possible. The eighth item in 
the questionnaire asked students about their comfort level with accessing the Internet 
(Figure 34). 
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N = 26 Mean = 6.77 Median = 7 

Minimum = 6 Maximum = 7 Mode = 7 

This distribution is very similar to responses to the identical question in the A-2 
questioimaire (Figure 11). As in responses to the same question in A-2, significant 
correlations are high and numerous (Table 34, compare to Table 13). On both occasions, 
when this question was asked, the highest correlations were Avith the comfort level with 
links, use of e-mail, and postings to forums. In this questionnaire there are some variables 
not present in the data analysis in the A-2 questioimaire such as the set of questions about 

J 

features used in the course (rules of engagement, module objectives, and on-line student 
tracking data). Some of the variables fi'om these sets also appeared to be statistically 
significant, including three indicators from student tracking. 
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Instrument 

Statement / Variable 

P 


Module 14 

12- ‘How would you rate your comfort level at this 
point with the links (i.e., additional web sites) 
feature being used to teach this course?’ 

0.794 


Module 14 

9 - ‘How would you rate your comfort level at this 
point with sending electronic mail?’ 

0.778 

< 0.001 

Module 14 

11 - ‘How would you rate your comfort level at this 
point with quiz feature being used to teach this 
course?’ 

0.695 

< 0.001 

Module 14 

10 - ‘How would you rate your comfort level at this 
point with posting comments to the Forums?’ 

0.656 

< 0.001 

Module 14 

13 - ‘Identifying elements of space systems that are 
vulnerable to disruption, degradation or destruction 
by adversaries.’ 

0.653 

<0.001 

Student 

tracking 

Number of visits to homepage 

0.475 

0.005 

Student 

tracking 

Number of articles read 

0.461 

0.010 

Module 14 

1 - ‘Identifying key elements of Space Control.’ 

0.470 

■ifilKI 

Student 

tracking 

Total number of hits 

0.425 

0.019 

Module 14 

5 - ‘Articulating how the concept of space control 
relates to Joint Vision 2020.’ 


0.035 


Table 34. Significant Correlations and Levels of Significance between the Eighth 
Statement and other Variables (Ordered by Level of Significance). 

The next question was about the comfort level with sending e-mail (Figure 35). 
Now all the students felt comfortable or extremely comfortable sending e-mail, which 
was not the case in the A-2 questionnaire (Figure 14), where three students expressed 
they felt somewhat (one student) or even extremely uncomfortable (two students). 
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Comfort Level with Sending E-mail 



Figure 35. Distribution of the Responses to the Ninth Statement. 

N = 26 Mean = 6.85 Median = 7 

Minimum = 6 Maximiun = 7 Mode = 7 

The dominant issues widi the highest correlations seem to be different than in the 
A-2 questionnaire (Table 35 below, compare with Table 15). In the latter case, the 
highest correlation was with the comfort level with word processing, which is not 
significant in this questionnaire. There are also some other variables that are statistically 
significant such as four correlations with on-line tracking indicators. Correlations with 
the on-line tracking data indicate that students more comfortable with sending e-mail tend 
to read more articles, visit the course homepage more often, post more articles, and in 
general visit on-line course material more fi'equently. Overall, comfort level with sending 
e-mail seems to be important for student achievement in the course. 
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The next question was: ‘How would you rate your comfort level at this point with 
posting comments to the Forums?’ Figure 36 shows the distribution of the responses. 
Twelve students (46.15%) felt ‘extremely comfortable’ with posting to the forums, two 
students (7.69%) felt ‘somewhat comfortable’, and two even ‘somewhat uncomfortable’. 
Compared to the distribution of the responses to a similar question in A-2 questionnaire 
(Figure 13), this distribution shows that the comfort level improved during the course. 


Instrument 

Statement / Variable 

P 

p value 

Module 14 

8 - ‘How would you rate your comfort level at this 
point with accessing the Internet?’ 

0.778 

<0.001 

Module 14 

13 - ‘Identifying elements of space systems that are 
vulnerable to disruption, degradation or destruction 
by adversaries.’ 

0.662 

<0.001 

Module 14 

11 - ‘How would you rate your comfort level at this 
point with quiz feature being used to teach this 
course?’ 

0.600 

0.001 

Module 14 

10 - ‘How would you rate your comfort level at this 
point with posting comments to the Forums?’ 

0.567 

0.003 

Module 14 

12- ‘How would you rate your comfort level at this 
point with the links (i.e., additional web sites) 
feature being used to teach this course?’ 



Module 14 

1 - ‘Identifying key elements of Space Control.’ 

0.494 

0.010 

Student 

tracking 

Number of articles read 

0.420 

0.021 

Student 

tracking 

Number of visits to homepage 

0.400 

0.029 

Student 

tracking 

Number or articles posted 



Student 

tracking 

Total number of hits 

0.387 

0.034 

Module 14 

5 - ‘Articulating how the concept of space control ' 
relates to Joint Vision 2020.’ 

0.388 

0.050 


Table 35. Significant Correlations and Levels of Significance between the Ninth 
Statement and Other Variables (Ordered by Level of Significance). 
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Similar items appear on the list of highly correlated variables, and correlations 
with five student tracking indicators are statistically significant also (Table 36 below, 
compare with Tables 14 and 19). 


Comfort Level with Posting Comments to the 
Forums 



Figure 36. Distribution of the Responses to the Tenth Statement. 

N = 26 Mean = 6.12 Median = 6 

Minimum = 3 Maximum = 7 Mode = 7 

A question on the comfort level with the quiz feature was next. The distribution of 
the responses (Figure 37) had much more responses about high comfort level than in the 
questionnaire A-2 (Figure 21). However, this change is not only due to the learning 
process and improving one’s own skills in the covurse, but also due to the fact that 
technical problems such as ‘bugs’ in software were fixed after the A-2 was administered. 

This variable has numerous significant correlations with other variables (Table 
37), including six student tracking indicators and final grades. The instructor did not 
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include quiz scores in the final grade, but the correlation, even though not high'2, is 
evident. Also, there are similarities to correlations in the A-2 question about the qmz 
feature. Compare it to Table 23. 


Instrument 

Statement / Variable 

P 


Module 14 

11 - ‘How would you rate your comfort level at this 
point with quiz feature being used to teach this 
course?’ 

0.687 

<0.001 

Module 14 

13 - ‘Identifying elements of space systems that are 
vulnerable to disruption, degradation or destruction 
by adversaries.’ 

0.669 

< 0.001 

Module 14 

8 - ‘How would you rate your comfort level at this 
point with accessing the Internet?’ 

0.656 

<0.001 

Module 14 

9 - ‘How would you rate your comfort level at this 
point with sending electronic mail?’ 

0.567 

0.003 

Module 14 

12- ‘How would you rate your comfort level at this 
point with the links (i.e., additional web sites) 
feature being used to teach this course?’ 

0.565 

0.003 

Student 

tracking 

Number of original posts 

0.485 

0.007 

Student 

tracking 

Total number of hits 

0.441 

0.015 

Student 

tracking 

Niunber or articles posted 

1 

0.416 

0.022 

Student 

tracking 

Number of visits to homepage 

0.406 

0.026 

Student 

tracking 

Number of articles read 

0.373 

0.043 


Table 36. Significant Correlations and Levels of Significance between the Tenth 
Statement and Other Variables (Ordered by Level of Significance). 


By ‘high’ correlation statisticians usually mean values over 0.75 or 0.80. 

108 
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Figure 37. Distribution of the Responses to the Eleventh Statement. 


N = 26 Mean = 6.35 Median = 7 

Minimum = 3 Maximum = 7 Mode = 7 

The question about the links feature has the highest percent of ‘extremely 
comfortable’ responses - 76.92% (Figure 38). Only one student felt ‘somewhat 
uncomfortable’ with the links used in the course. Similar responses about this feature 
were in questionnaire A-2 (Figure 17), but with one ‘extremely uncomfortable’ response. 
There is only one very high correlation (Table 38) with the comfort level in accessing the 
Internet. This is logical because problems with access certainly cause many other 
problems in using the course features, which includes links, but the other five correlations 
were statistically significant also. Interestingly, the number of significant correlations was 
higher in A-2 questioimaire (Table 22). 
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Instrument 

Statement / Variable 

P 


Module 14 

8 - ‘How would you rate your comfort level at this 
point with accessing the Internet?’ 

0.695 

<0.001 

Module 14 

10 - ‘How would you rate your comfort level at this 
point with posting comments to the Forums?’ 



Module 14 

9 - ‘How would you rate your comfort level at this 
point with sending electronic mail?’ 

0.600 

0.001 

Student 

tracking 

Total number of hits 

0.581 

0.001 

Module 14 

13 - ‘Identifying elements of space systems that are 
vulnerable to dismption, degradation or destruction 
by adversaries.’ 

0.542 

0.004 

Student 

trackiiig 

Number of articles posted 

0.497 

0.005 

Student 

tracking 

Number of articles read 



Student 

tracking 

Number of visits to homepage 

0.449 

0.013 

Module 14 

12- ‘How would you rate your comfort level at this 
point with the links (i.e., additional web sites) 
feature being used to teach this course?’ 

0.475 

IQQI 


FINAL GRADE 

0.465 

■iftngB 

Student 

tracking 

Number of original posts 

0.427 

0.019 

Module 14 

1 - ‘Identifying key elements of Space Control.’ 

0.440 

0.024 

Student 

tracking 

Number of follow-up posts 

0.411 

0.024 


Table 37. Significant Correlations and Levels of Significance between the Eleventh 
Statement and Other Variables (Ordered by Level of Significance). 
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Figure 38. Distribution of the Responses to the Twelfth Statement. 

N = 26 Mean = 6.62 Median = 7 

Minimum = 3 Maximiim = 7 Mode = 7 

The students responded to the thirteenth question about the comfort level with 
word processing in a similar manner as the comfort level with links used in the course. 
The vast majority (twenty students or 76.92%) felt extremely comfortable, which is what 
is expected from students at NFS (Figure 39). This distribution shows slightly higher 
student comfort level than results in the second questionnaire (compare Figure 7). Again, 
there were many significant correlations with other variables such as those asking about 
comfort level with forums, finals grades, e-mail, accessing the Internet etc., and with five 
student tracking indicators (Table 39). The comfort level with word processing seems to 
be very indicative of students’ ‘broad technical proficiency’, which is so important in a 
DL environment. A similarity about correlations between such variables is present in the 
A-2 questionnaire (Table 14). 
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Instrument 

Statement 

P 

p value 

Module 14 

8 - ‘How would you rate your comfort level at this 
point with accessing the Internet?’ 

0.794 

< 0.001 

Module 14 

10 - ‘How would you rate your comfort level at this 
point with posting comments to the Forums?’ 

0.565 

0.003 

Module 14 

9 - ‘How would you rate your comfort level at this 
point with sending electronic mail?’ 

0.531 

0.005 

Module 14 

11 - ‘How would you rate your comfort level at this 
point with quiz feature being used to teach this 
course?’ 



Module 14 

13 - ‘How would you rate your comfort level at this 
point with word processing?’ 

0.445 

0.023 

Module 14 

7 - ‘Identifying elements of space systems that are 
vulnerable to disruption, degradation or destruction 
by adversaries.’ 

0.398 

0.044 


Table 38. Significant Correlations and Levels of Significance between the Twelfth 
Statement and Other Variables (Ordered by Level of Significance). 
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Figure 39. Distribution of the Responses to the Thirteenth Statement. 


N = 26 Median = 7 Minimum = 5 

Maximum = 7 Mode = 7 
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Instrument 

Statement / Variable 

P 

p value 

Module 14 

10 - ‘How would you rate your comfort level at this 
point with posting comments to the Forums?’ 

0.669 

<0.001 

Module 14 

9 - ‘How would you rate your comfort level at this 
point with sending electronic mail?’ 

0.662 

<0.001 

Module 14 

8 - ‘How would you rate your comfort level at this 
point wi& accessing the Internet?’ 

0.653 

<0.001 

Module 14 

11 - ‘How would you rate your comfort level at this 
point with quiz feature being used to teach this 
course?’ 

0.542 

0.004 

Student 

tracking 

Number or articles posted 

0.501 

0.005 

Student 

tracking 

Number of articles read 

0.478 

0.008 


FINAL GRADE 

0.441 

0.015 

Module 14 

12- ‘How would you rate your comfort level at this 
point with the links (i.e., additional web sites) 
feature being used to teach this course?’ 

0.445 

0.023 

Student 

tracking 

Number of original posts 

0.412 

0.024 

Student 

tracking 

Number of follow-up posts 

0.407 

0.026 

Student 

tracking 

Total munber of hits 

0.404 

0.027 


Table 39. Significant Correlations and Levels of Significance between the Thirteenth 
Statement and odier Variables (Ordered by Level of Significance). 


The last question in the Module 14 questionnaire was open-ended, thereby giving 
students an opportunity to add comments. The comments varied fi'om simple and positive 
to rather complex. The students also gave suggestions for course improvements. Here are 
the comments grouped into five types: 

Positive comments 

• This course is a great for keeping up on current events, as well as keeping 
our interest in the topic by linking us to good web resources 

• I enjoyed the course! I look forward to seeing more courses. I feel this is 
the beginning of some great opportunities for troops who would not 
otherwise be able to take classes due to location or OPTEMPO. 
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This was my first online course and I truly enjoyed it On-line courses take 
a great deal of personal commitment and focus. Providing an interesting 
subject and stimulating discussion helps to increase participation. 

Having no prior courses in space systems, I did not know what to expect. I 
am amazed at how much our military (as well as everyday life) is 
dependent on space systems. This especially comes as a surprise having 
been on 4 WESTPAC deployments and not being aware of what's going 
on "up there". It all seemed so transparent... not anymore. I appreciate this 
great course, which will ensure I will continue to educate myself about 
space systems. 

Honestly, if I was a space guy, I might not have gotten a great deal for this 
course and wouldn't have been challenged. As it is. I’m an ITM guy and 
feel that I received a great overview of Space, the systems involved, how 
the systems relate to the military, etc. I’ll admit it; I actually enjoyed the 
course and looked forward to reading the articles and postings. This 
definitely would not have been the case if I’d been forced into learning a 
lot of math that I would never use again (and would thus drop out of my 
brain's RAM as soon as the quarter was over). 

As I stated in my written SOF, had testing occurred weekly, it probably 
would have kept me in the books/readings with more consistency. The 
readings were very interesting...much more so than the text.... Some of the 
text concepts (orbits) were difficult to understand without direct classroom 
instruction... Overall, a good experience. 


Positive comments, mentioning some problems 

Although I enjoyed the flexibility of the on-line course, I am frustrated 
with the content. There were no technical discussions or material provided 
in the forums or quizzes. The required underst andin g of the material was 
rudimentary at best. If this was meant as an introductory course, then it 
met the requirement. If it was meant as a technical course, then it fell way 
short. 

This course is a good example of what you can do in an Internet course. I 
thought the forum played a good role in student interaction. The loss of the 
ability to quiz on later sections did tend to demotivate. I know it should be 
more than grades and testing, but with our schedules, you do what you are 
tested on. All arormd the course was a good experience. 

Outstanding links were provided to support objectives of each module. 
Very thoughtful, innovative, and interesting presentations. The class 
seemed to get a little off-track when the system kept going down, but 
made a strong finish. System reliability is a problem that needs to be 
addressed. Perhaps a backup server at a different location. I really enjoyed 
this class. 



• I think there is still plenty of room for improvement Network downtime 
and glitches interrupted the flow of the course. I believe the Quizzes for 
each module are a necessity. While they are a pain in the rear, they really 
force you to keep up with the material...once the quizzes tapered off... It is 
easy to get behind. The quizzes don't have to be big...just enough to keep 
you honest Other than that I think the course has great potential for 
learning. 


Some ideas and suggestions 

• Would like to see a “What’s New” page every time I log on some can 
instantly see what is happening relative to bug fixes, or new 
enhancements, etc. Also would like an email feature that would let me 
know when someone replied to a forum posting of my own. 

• I think there should be quizzes after each module to keep us on our toes! 
Aside from that, it was a very interesting course. 

• I think that forums every week, in small groups that change, would have 
drastically increased my consistent participation in the course. I would 
even say that part of the individual grade for a student would be made up 
of the average grade of all the groups he or she belonged to. Nothing like a 
little peer-pressure to get a group to encourage itself (member to member 
e-mails or whatever). 


Skeptics 

• An on-line course, will never provide the same quality of instruction as an 
actual in class course. I do not feel like I got more than a cursory 
knowledge of die material covered. The on-line format is adequate for 
training type courses- not education. To improve the class, there should 
be at least one class meeting per week (with VTC if required). All 
modules should be available at all times in order to work at your own pace 
and get ahead when you can. In the future, I will be much more less 
inclined to take an on-line class. If this is die wave of the future, our 
educational system is in bigger trouble than we think. 

• I felt I would have gotten more from a traditional course. I understand 
there were some technical problems with the course but I believe the face- 
to-face contact, at lease occasionally is really necessary. 


Complex comments 

The course was difficult to complete online. Classroom interface was 
needed. I think may be adding a Chat Room and class meeting time would 
help get a better understanding of what was required. The emails were 
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great help to keep me on track. However, the emails were late in support 
of the modules. The modules were too dynamic for me to follow. I think 
the webs site should put old modules or incorrect data on the web site. 
Most of the Qui 2 zes didn't work correctly. My quiz grade is still messed 
up. The interface with a keyboard is slow and painful. Spell check and 
grammar check would help. I did learn a lot about space systems from the 
links. At this point I would not hire a person with a degree earn online. 
The student work is hard to verify. Think of adding a private chat room for 
instructor to student test. Online Questions could be asked in a private 
chat as a test question. 

• I would have enjoyed this class more if it were my only class. It was a 
serious time HOG when it came to posting messages. What you could do 
in 2 minutes discussing all the point, it would take 45 minutes to an hour 
to write a well-articulated point paper... I think messages should be limited 
to 7/8 sentences max. Some people wrote books, which subconsciously 
made the stakes higher for everyone in trying to write something unique. 
Format of message post...to aid reading, users must write 3/4 sentences 
and put a break and start on the next line. This should be in the rules of 
engagement. Too many posts ran forever all together, and made it difficult 
to read. The whole forum message board is not extremely intuitive. I have 
it figured out but there are a lot of menu items. Unsure as to you could do 
to improve. Personally, I liked the online quizzes. Learning does take 
place when taking them. Bugs need to be worked out. Frustrating when the 
quizzes did not score correctly initially. 

• Strong points: 

o The information provided on the web site was excellent. The links 
were informative and rich with both interesting and relevant 
information. Most definitely, the best part of the course, 
o The forums allowed for an exchange of ideas. It's a good way to 
learn. 

• Weak points: 

o I could have done without the textbook. The website provided all 
the book had and more. I really don't think there’s a need for a 
textbook. 

o The forums weren't really a hotbed of new ideas or even an 
exploration of old ideas. In the future, some method of more 
regular interaction should be developed. 

• Overall: 

o It is very obvious that a great deal of work and time went into 
developing this course. I think many of the bugs experienced 
during this quarter for this course were temporary. This will be one 
heck of a course (on-line or otherwise) when everything is up and 
r unning ! 
o 
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Module 14 Questionnaire - Summary 

This on-line questionnaire was administered after module 14. In total 15 modules 
were covered in the course. This questionnaire was not anonymous; twenty-six students 
completed it. 

There were three groups of questions/statements. The first seven were designed 
by the instructor to see how well the module 14 learning objectives were being met. The 
next six questions asked the students about their comfort level with relevant ‘technical’ 
issues and features used in the course such as accessing the Internet, sending e-mail, 
posting comments to forums, completing the quiz feature used in the course, using the 
links feature and lastly, using word processing. The last question was open-ended; 
students had an opportunity to add any comments or additional thoughts. 

Responses to the first seven statements were predo minantl y positive, but with 
slightly different distributions. Some statements had a very high number of ‘agree’ and 
‘strongly agree’ responses, and some others had a significant number of ‘disagree’ 
responses. In general, the students responded that all module objectives were more or less 
met to a very satisfactory degree. 

Responses about comfort levels with features mentioned above are very similar to 
the reqwnses to the corresponding questions in the Second questionnaire (A-2), but with 
increased overall comfort level. Student on-line tracking indicators and some items from 
the First questionnaire had significant correlations with several questions in this 
questionnaire, like questions about comfort level with accessing the Internet, with 
sending e-mail, with posting to forums, etc. 
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The student comments provided many interesting comments, which were grouped 
into the following content types: positive comments (six students); positive comments 
with mentioning some problems (four); ideas and suggestions (three); complex comments 
(three); and ‘skeptics’ (two students). Seven students did not add any comments. 

4. Final Survey 

This on-line questionnaire was administered after the course was completed, after 
the final exam. However, it was administered before the she instructor gave final grades 
to the students. It was anonymous. Twenty-three students completed it. 

The thirty-one items included: 

• Questions reflecting course objectives 

• Two questions on student self-confidence in certain activities relevant to 
the course 

• Several questions on the usefulness of diflferent features used in the course 
in terms of helping the student to understand the course concepts, 
objectives, and principles; four questions asking the students how much 
they liked some of the activities and materials in the course 

• Four questions about interactivity between the students and the instructor 

• Some other specific questions included open-ended questions 

• Long-Dziuban’s Reactive Behavior Protocol 


There were five open-ended questions in this questionnaire. 

The first question asked the students the following: ‘As a result of your overall 
experience in this course, do you feel that you are able to articulate how physical 
influences on space systems impact our ability to use those systems in mili tary 
operational situations?’ Figure 40 shows how the students responded. 
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Figure 40. Distribution of the Responses to the First Question. 

N = 23 Mean = 5.65 Median = 6 

Minimum = 3 Maximum = 7 Mode = 6 


Most of the students (nineteen out of twenty-three or 82.61%) clearly agreed 
they’ve become able to articulate space system issues. Responses to this question 
significantly correlate with affective reactions, the module readings usefulness question, 
self-confidence in discussing and in accessing resources, and the quality of interaction 
between the students and the instructor (Table 40). Seven correlations have p under 0.01, 
i.e., 1%. 

The second question was: ‘As a result of your experiences in this course, are you 
able to articulate how political, organizational and economic influences impact how we 
use space systems in military operational settings?’ Responses to this question (Figure 
41) were similar to the first one, but all were positive and only one was neutral. Seven 
correlations were statistically significant (Table 41). 
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Statement / Variable 

P 

p value 

17 - ‘How did you like the website links that were provided?’ 

0.629 

0.001 

10 - ‘How useful were the on-line Module readings in helping you 
to understand the course concept... ’ 

0.596 

0.003 

3 - ‘Do you feel that you are able to articulate how commercial 
space systems will impact military operations?’ 

0.592 

0.003 

22 - ‘Did you view the flexibility of this on-line course as 
beneficial to your ability to succeed in this class?’ 

0.585 

0.003 

6 - ‘I feel confident in ihy ability to get involved in open 
discussions with my peers to weigh issues related to using space 
systems in operational settings.’ 

0.550 

0.007 

2 - ‘As a result of your experiences in this course, are you able to 
articulate how political, organizational and economic influences 
impact how we use space systems in military operational settings? 

0.538 

0.008 

5 - ‘I feel confident in my ability to access appropriate resources, 
such as subject matter experts, documentation of websites, or late 
breaking news for evaluating uses of space systems in operational 

situations.’ 

0.540 


26 - ‘Compare the quality of interaction between yourself and the 
instructor in this on-line class against what you would have 
expected in a similar ‘face-to-face’ class.’ 

0.429 



Table 40. Significant Correlations and Levels of Significance between the First 

Question and Other Variables (Ordered by Level of Significance). 


"Articulate Political, Organizational, and Economic 
Influences on Use of Space Systems" 



Figure 41. Distribution of the Responses to the Second Question. 

N = 23 Mean = 5.83 Median = 6 

Minimum = 4 Maximum = 7 Mode = 6 
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Statement / Variable 

P 

p value 

3 - ‘Do you feel that you are able to articulate how commercial 
space systems will impact military operations?’ 

0.782 

<0.001 

20 - ‘I am certain that I am able to synthesize information fi'om a 
variety of sources to enable me to use space systems in operational 

situations.’ 

0.634 

0.001 

6 - ‘1 feel confident in my ability to get involved in open 
discussions with my peers to weigh issues related to using space 
systems in operational settings.’ 

0.633 

0.001 

10 - ‘How useful were the on-line Module readings in helping you 
to understand the course concept... ’ 

0.585 

0.003 

1 - ‘. .Do you feel that you are able to articulate how physical 
influences on space systems impact our ability to use those systems 
in military operational situations?’ 

0.538 

0.008 

5 - ‘I feel confident in my ability to access appropriate resources, 
such as subject matter experts, documentation of websites, or late 
breaking news for evaluating uses of space systems in operational 

situations.’ 

0.494 

0.016 

11 - ‘How useful were the linked websites in helping you to 
understand the coinse concepts, objectives and principles?’ 

0.476 

0.022 


Table 41. Significant Correlations and Levels of Significance between the Second 
Question and Other Variables (Ordered by Level of Significance). 


The third question was: ‘As a result of your experiences in this course, are you 
able to articulate how political, organizational and economic influences impact how we 
use space systems in military operational settings?’ Figure 42 shows how the responses 
were distributed. Ten students (43.48%) responded with ‘strongly agree’, eight (34.78%) 
with ‘agree’, and five (21.74%) with ‘slightly agree’. None were neutral or disagreed. Six 
correlations were statistically significant (Table 42). The listed statements/questions dealt 
with the issues of different student abilities and with students’ view of the usefulness of 
module readings. 
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"Articulate Impact of Commercial Space Systems on 
Military Operations" 



Figure 42. Distribution of the Responses to the Third Question. 

N = 23 Mean = 6.21 Median = 6 

Minimum = 4 Max = 7 Mode = 7 


The fourth question asked students: ‘Was the final exam useful in helping you to 
synthesize the concepts learned in the coinse?’ Figure 43 shows the responses. Four 
students (17.39%) thought the exam was not useful. Only three (13.04%) believed the 
exam was extremely useful. Table 43 shows the variables that had significant correlations 
to this question responses. The variables listed in the table include three ‘affective 
reaction’ (liking) type questions and one question on course interactivity. 
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Statement / Variable 

P 

p value 

2 - ‘As a result of your experiences in this course, are you able to 
articulate how political, organizational and economic influences 
impact how we use space systems in military operational settings? 

0.782 

<0.001 

20 - ‘I am certain that I am able to synthesize information firom a 
variety of sources to enable me to use space systems in operational 

situations.’ 

0.605 

0.002 

6 - ‘I feel confident in my ability to get involved in open 
discussions with my peers to weigh issues related to using space 
systems in operational settings.’ 

0.602 

0.002 

1 - ‘. .Do you feel that you are able to articulate how physical 
influences on space systems impact our ability to use those systems 
in military operational situations?’ 

0.592 


10 - ‘How useful were the on-line Module readings in helping you 
to understand the course concept... ’ 

0.588 

0.003 

5 - ‘I feel confident in my ability to access appropriate resources, 
such as subject matter experts, documentation of websites, or late 
breaking news for evaluating uses of space systems in operational 

situations.’ 

0.557 

0.006 


Table 42. Significant Correlations and Levels of Significance between the Third 
Question and Other Variables (Ordered by Level of Significance). 


"Was the Final Exam Useful in Helping You to 
Synthesize the Concepts...?" 



Figure 43. Distribution of the Responses to the Fourth Question. 
N = 23 Mean = 5.13 Median = 5 

Minimum = 2 Max = 7 Mode = 6 
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Statement / Variable 

P 

p value 

10 — ‘How useful were the on-line Module readings in helping you 
to understand the course concept.., ’ 

0.537 

0.008 

25 - ‘Compare the quality of interaction between yourself and the 
instructor in this on-line class against what you would have 
expected in a similar ‘face-to-face’ class.’ 

0.523 

0.011 

8 - ‘How useful were guidelines provided at the course homepage 
for posting substantive responses to the Forums?’ 



23 - ‘Do you feel that the flexibility of this class helped you 
succeed in other classes that you were taking in the same quarter?’ 

0.503 

0.014 

24 - ‘Compare the amount of interaction you had with the 
instructor for this on-line course as compared with a similar 
resident ‘face-to-face’ course.’ 

0.456 

0.029 

17 — ‘How did you like the website links that were provided?’ 

0.453 

WB5W 

16 - ‘How did you like the Module readings?’ 



15 - ‘How did you like the textbook reading assignments?’ 

0.428 


12 — ‘How useful were the Forum exercises in helping you to 
understand the subject matter?’ 

0.422 

0.045 


Table 43. Significant Correlations and Levels of Significance between the Fourth 
Question and Other Variables (Ordered by Level of Significance). 

The fifth question addressed the students’ confidence: ‘I feel confident in my 
ability to access appropriate resources,- such as subject matter experts, documentation of 
websites, or late breaking news for evaluating uses of space systems in operational 
situations.’ Most of the students (nine or 39.13%) chose the response ‘agree’, and none 
disagreed (Figure 44). The highest correlation this variable had was with another question 
about confidence (Table 44). In total, eight correlations were statistically significant. 
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Figure 44. Distribution of the Responses to the Fifth Question. 

N = 23 Mean = 5.83 Median = 6 

Minimum = 4 Max = 7 Mode = 6 


The sixth question asked: T feel confident in my ability to get involved in open 
discussions with my peers to weigh issues related to using space systems in operational 
settings.’ Figure 45 shows how the responses were distributed and a slight agreement 
(nine responses or 39.13%) with the statement dominated. Table 45 lists seven variables 
that significantly correlate with the responses to this statement. 
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Statement / Variable 

P 

p value 

6 - ‘I feel confident in my ability to get involved in open 
discussions with my peers to weigh issues related to using space 
systems in operational settings.’ 

0.736 

< 0.001 

3 - ‘Do you feel that you are able to articulate how commercial 
space systems will impact military operations?’ 

0.557 

0.006 

1 - ‘. .Do you feel that you are able to articulate how physical 
influences on space systems impact our ability to use those systems 
in military operational situations?’ 

0.540 

0.008 

20 - ‘I am certain that I am able to synthesize information fi-om a 
variety of sources to enable me to use space systems in operational 

situations.’ 

0.509 

0.013 

2 - ‘As a result of your experiences in this course, are you able to 
articulate how political, organizational and economic influences 
impact how we use space systems in military operational settings? 

0.494 

0.016 

22 - ‘Did you view the flexibility of this on-line course as 
beneficial to your ability to succeed in this class?’ 

0.482 

0.020 

10 - ‘How useful were the on-line Module readings in helping you 
to tmderstand the course concept... ’ 

0.444 

0.034 

16 - ‘How did you like the Module readings?’ 

0.436 

0.038 


Table 44 Significant Correlations and Levels of Significance between the Fifth 
Question and Other Variables (Ordered by Level of Significance). 


"I Felt Confident in My Ability to Get Involved in 
Open Discussions with My Peers..." 


10 

9 

8 

7 

Number of ® 

students 4 
3 
2 
1 
0 




Figure 45. Distribution of the Responses to the Sixth Question. 
N = 23 Mean =5.65 Median = 6 


Minimum = 4 


Max = 7 
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Mode = 5 
















Statement / Variable 

P 

p value 

5 - ‘I feel confident in my ability to access appropriate resources, 
such as subject matter experts, documentation of websites, or late 
breaking news for evaluating uses of space systems in operational 

situations.’ 

0.736 

<0.001 

2 - ‘As a result of your experiences in this course, are you able to 
articulate how political, organizational and economic influences 
impact how we use space systems in military operational settings? 

0.633 

0.001 

3 - ‘Do you feel that you are able to articulate how commercial 
space systems will impact military operations?’ 

0.602 

0.002 

10 - ‘How useful were the on-line Module readings in helping you 
to understand the course concept... ’ 

0.564 

0.005 

22 - ‘Did you view the flexibility of this on-line course as 
beneficial to your ability to succeed in this class?’ 

0.555 

0.006 

1 - ‘. .Do you feel that you are able to articulate how physical 
influences on space systems impact our ability to use those systems 
in military operational situations?’ 

0.550 

0.007 

20 - ‘I am certain that I am able to synthesize information fi'om a 
variety of sources to enable me to use space systems in operational 

situations.’ 

0.478 

0.021 


Table 45. Significant Correlations and Levels of Significance between the Sixth 
Question and Other Variables (Ordered by Level of Significance). 


The seventh question and the distribution of the responses are in Figure 46. Asked 
about usefulness of the guidelines on the course homepage, few students (two or 8.70%) 
chose the response ‘useless’, but a majority of the students (twenty students or 86.96%) 
saw the guidelines as useful. Only one correlation was significant (Table 46), and that 
was with responses to the question asking about guidelines. 
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"How Useful Were the Guidelines Provided in the 
Course Homepage for Your Abiiity to Succeed?" 



Figure 46. Distribution of the Responses to the Seventh Question. 


N = 23 Mean = 5,61 Median = 6 

Minimum = 2 Max = 7 Mode = 6 


Statement / Variable 

P 


8 - ‘How useful were guidelines provided at the course homepage 
for posting substantive responses to the Forums?’ 

0.711 

<0.001 


Table 46. Significant Correlation and Level of Significance between the Seventh 

Question and Another Variable. 

Student responses to the second question about the guidelines (‘How useful were 
guidelines provided at the course homepage for posting substantive responses to the 
Forums?’) distributed differently than the first question (Figure 47), with more neutral 
responses (five or 21.74%) and a higher frequency of ‘useful’ responses (thirteen or 
56.53%). Four correlations appeared to be significant (Table 47). Interestingly, the 
responses to questions about module readings, quizzes, and final exams have significant 
correlations with responses to this question but to none of the other variables. This is 
difficult to explain. 
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"How Useful Were the Guidelines for Posting 
Substantive Responses to the Forums?" 



Figure 47. Distribution of the Responses to the Eighth Question. 

N = 23 Mean = 5.30 Median = 6 

Minimum = 2 Max = 7 Mode = 6 


Statement / V ariable 

P 

p value 

7 - ‘How useful were the guidelines provided in the course 
homepage to your ability to succeed?’ 

0.711 

<0.001 

16 - ‘How did you like the Module readings?’ 

0.516 

0.012 

4 - ‘Was the final exam useful in helping you to synthesize the 
concepts learned in the course?’ 

0.506 

0.014 

13 - ‘How useful were the Quizzes (2,3/4/5,6/7) for providing 
feedback in your understanding of the material for those Modules?’ 

0.500 

0.015 


Table 47. Significant Correlations and Levels of Significance between the Eighth 
Question and Ollier Variables (Ordered by Level of Significance). 

I 

The ninth question asked students about the usefulness of the textbook: ‘How 
useful was the textbook in helping you understand the course concepts, objectives and 
principles?’ Figure 48 shows the distribution of the student responses. Most of the 
students (twenty-one students or 91.30%) saw the textbook as useful, but there was one 
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response ‘useless’, and one neutral response. Only one correlation was significant (Table 
48). 


"How Useful Was the Textbook in Helping You 
to Understand the Course Concepts...?" 

14 
12 
10 

Number of 8 

students 6 
4 
2 
0 




Figure 48. Distribution of the Responses to the Ninth Question. 

N = 23 Mean = 6.13 Median = 6 

Minimum = 2 Max = 7 Mode = 6 


Statement / Variable 

P 


15 - ‘How did you like the textbook reading assignments?’ 

0.636 

0.001 


Table 48. Significant Correlation and Level of Significance between the Ninth 

Question and Another Variable. 

Most students (fourteen students or 60.87%) saw the on-line module readings as 
‘extremely useful’ (Figure 49). Ten correlations were significant. The correlations and 
significance levels are in Table 49. 


130 









Figure 49. Distribution of the Responses to the Tenth Question. 


N = 23 Mean = 6.57 Median =7 

Minimum = 7 Max = 7 Mode = 7 

The fifth question asked students: ‘How useful were the linked websites in 
helping you to imderstand the course concepts, objectives, and principles?’ Most students 
chose to respond with ‘useful’ (eleven students or 47.83%), and slightly less (ten or 
43.48%) with ‘extremely useful’ (Figure 50). 

Only three correlations were significant (Table 50). The highest correlation was 
with another question related to liking the links feature the course provided. This 
correlation of approximately 0.56 means that a utility assessment ( thinking of usefulness 
of a feature) and an affective reaction (liking it) have a significant association but are not 
identical. Otherwise the correlation would be much higher, close to 1. Two other 
correlations are difficult to interpret. They are correlations with the student responses to 

statements/questions related to several of the course learning objectives. 
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Statement / Variable 

P 

p value 

17 - ‘How did you like the website links that were provided?’ 

0.644 

HE SB 

1 - ‘. .Do you feel that you are able to articulate how physical 
influences on space systems impact our ability to use those systems 
in military operational situations?’ 

0.596 

0.003 

3 - ‘Do you feel that you are able to articulate how commercial 
space systems will impact military operations?’ 

0.588 

0.003 

2 - ‘As a result of your experiences in this course, are you able to 
articulate how political, organizational and economic influences 
impact how we use space systems in military operational settings? 

0.585 

0.003 

6 - ‘I feel confident in my ability to get involved in open 
discussions with my peers to weigh issues related to using space 
systems in operational settings.’ 

0.564 

0.005 

16 - ‘How did you like the Module readings?’ 

0.541 

0.008 

4 - ‘Was the final exam useful in helping you to synthesize the 
concepts learned in the course?’ 

0.537 

0.008 

23 - ‘Do you feel that the flexibility of this class helped you 
succeed in other classes that you were taking in the same quarter?’ 

0.536 

0.008 

22 - ‘Did you view the flexibility of this on-line course as 
beneficial to your ability to succeed in this class?’ 

0.512 

0.012 

5 - ‘I feel confident in my ability to access appropriate resources, 
such as subject matter experts, documentation of websites, or late 
breaking news for evaluating uses of space systems in operational 

situations.’ 

0.444 

0.034 


Table 49. Significant Correlations and Levels of Significance between the Tenth 
Question and Other Variables (Ordered by Level of Significance). 
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"How Useful Were the Linked Websites in 


Helping You to Understand the Course 
Concepts...?" 



Figure 50. Distribution of the Responses to the Eleventh Question. 

N = 23 Mean = 6.35 Median = 6 

Minimum = 5 Max = 7 Mode = 6 


Statement / Variable 

P 

p value 

17 - ‘How did you like the website links that were provided?’ 

0.557 


2 - ‘As a result of your experiences in this course, are you able to 
articulate how political, organi 2 ational and economic influences 
impact how we use space systems in military operational settings? 

0.476 

0.022 

20 -.‘I am certain that I am able to synthesize information fi'om a 
variety of sources to enable me to use space systems in operational 

situations.’ 

0.475 

0.022 


Table 50. Significant Correlations and Levels of Significance between the Eleventh 

Question and Other Variables (Ordered by Level of Significance). 

The twelfth question in the Final Survey questionnaire was about the usefulness 
of the forum exercises. Responses to this question vary fi-om ‘useless’ (three students or 
13.04%) to ‘extremely usefiil’ (one student or 4.35%), with responses ‘useful’ 
dominating (eight responses or 34.78%). Variability of the responses is larger than in the 

previous questions firom this set (Figure 51). Similar to file eleventh question, this 
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question had two correlations. The highest correlation was with the response to the 
question about liking the forums (Table 51). 


"How Useful Were the Forum Exercises in Helping 

You to Understand the Subject Matter?" 

Q __ __ 

V/ 

7. 
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Number of ^ 



students , 
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n. 






Figure 51. Distribution of the Responses to the Twelfth Question. 


N = 23 Mean = 4.65 Median = 5 

Minimxmi = 2 Max = 7 Mode = 6 


Statement / Variable 

P 


14 - ‘How did you like the Forum assignments?’ 

0.620 

0.002 

4 - ‘Was the final exam useful in helping you to synthesize the 
concepts learned in the course?’ 

0.422 

0.045 


Table 51. Significant Correlations and Levels of Significance between the Twelfth 

Question and Other Variables (Ordered by Level of Significance). 

The instructor recommended the next interesting question to determine if the 

quizzes in the course served the purpose she had intended: as feedback to the students. 

Figure 52 shows the distribution of the responses. Most of the students (seventeen or 

73.91%) saw the quizzes as more or less useful as feedback, but four (17.39%) had the 

opposite opinion. Two (8.70%) chose neutral responses. Five correlations were 

significant, but none above 0.50. It is difficult to understand why this variable correlates 
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with variables dealing with module readings, interactivity between the students, and the 
flexibility that helped in succeeding in other classes (Table 52). 


"How Useful Were the Quizzes for Providing 
Feedback in Your Understanding of the Material...?" 



Figure 52. Distribution of the Responses to the Thirteenth Question. 

N = 23 Mean = 4.78 Median = 6 

Minimum = 2 Max = 7 Mode = 6 


Statement / Variable 

P 

p value 

8 - ‘How useful were guidelines provided at the course homepage 
for posting substantive responses to the Forums?’ 

0.500 

0.015 

23 - ‘Do you feel that the flexibility of this class helped you 
succeed in other classes that you were taking in the same quarter?’ 

0.492 

0.017 

27 — ‘Compare the quality of interaction you had with other 
students for this on-line course as compared with a similar resident 
‘face-to-face’ course.’ 

0.492 

0.017 

16 - ‘How did you like the Module readings?’ 

0.463 

0.026 

17 - ‘How did you like the website links that were provided?’ 

0.442 

0.035 


Table 52. Significant Correlations and Levels of Significance between the Thirteenth 
Question and Other Variables (Ordered by Level of Significance). 

The next question asked the students how they liked forum assignments (Figure 

53). Most of them ‘slightly liked’ (eight students or 34.78%) and ‘liked’ (six students 
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26.09%) them, but four students (17.39%) gave negative responses. Five (21.74%) gave 
neutral responses. Only two correlations were significant; one with responses to the 
question about the usefulness of forum exercises which is a very logical relationship, and 
another, about the amount of interactivity between the students and the instructor. The 
students, who saw the forums as useful, liked them. Those who thought that the course 
provided a satisfactory amoimt of interaction compared to face-to-face classes with the 
instructor tended to like forum assignments more than the students unsatisfied with the 
amount of interaction (Table 53). 



Figure 53. Distribution of the Responses to the Fourteenth Question. 

N = 23 Mean = 4.56 Median = 5 

Minimum = 1 Max = 6 Mode = 5 
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Statement / Variable 

P 

p value 

12 - ‘How useful were the Forum exercises in helping you to 
understand the subject matter?’ 

0.620 


24 - ‘Compare the amount of interaction you had with the 
instructor for this on-line course as compared with a similar 
resident ‘face-to-face’ course.’ 

0.487 

0.018 


Table 53. Significant Correlations and Levels of Significance between the 
Fourteenth Question and Other Variables (Ordered by Level of Significance). 


Figure 54 shows the distribution of student responses about liking the textbook 
assignments. Most of them (eleven students or 47.83%) only ‘slightly liked’ them, a 
substantial number (seven students or 30.43%) ‘liked’ them, but two students (8.70%) 
disliked them, while three (13.04%) were neutral. Table 54 shows three significant 
correlations this variable had with responses to the question about the usefulness of the 
textbook, usefulness of the final exam, and about students’ feeling about how well this 
this course flexibility helped them for success in other courses. The last two correlations 
are not easy to explain, while the first relationship is very understandable. 


"How did you Like the Textbook Reading 
Assignments?" 


Number of 
students 




Figure 54. Distribution of the Responses to the Fifteenth Question. 
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N = 23 
Minimum = 1 


Mean = 4.91 
Max= 6 


Median = 5 
Mode = 5 


Statement / Variable 

P 

p value 

9 - ‘How useful was the textbook in helping you imderstand the 
course concepts, objectives and principles?’ 

0.636 

0.001 

4 - ‘Was the final exam useful in helping you to synthesize the 
concepts learned in the course?’ 

0.428 

0.042 

23 - ‘Do you feel that the flexibility of this class helped you 
succeed in other classes that you were taking in the same quarter?’ 

0.425 

0.043 


Table 54. Significant Correlations and Levels of Significance between the Fifteenth 
Question and Other Variables (Ordered by Level of Significance). 


'How did You Like the Module Readings?' 













iSS' 





Figure 55. Distribution of the Responses to the Sixteenth Question. 


N = 23 

Minimum = 2 


Mean = 5.34 
Max = 6 


Median = 6 
Mode = 6 


The next question asked the students how much they liked the modiile readings. 
Over 60% of the students chose response ‘liked’, and almost all others ‘slightly liked’ the 
readings(Figure 55). Ten correlations were significant, starting with questions on liking 
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website links, beneficial flexibility of the course, and usefulness of the module readings 
(Table 55). Other questions included issues like amount of interactivity with other 
students, quizzes, and final exams. 


Statement / Variable 

P 

p value 

17 - ‘How did you like the website links that were provided?’ 

0.632 

0.001 

22 - ‘Did you view the flexibility of this on-line course as 
beneficial to your ability to succeed in this class?’ 

0.590 

0.003 

10 - ‘How useful were the on-line Module readings in helping you 
to understand the course concept... ’ 

0.541 

0.008 

26 - ‘Compare the amount of interaction you had with other 
students for this on-line course as compared with a similar resident 
‘face-to-face’ course.’ 

0.539 

0.008 

8 - ‘How usefiil were guidelines provided at the course homepage 
for posting substantive responses to the Forums?’ 

0.516 

0.012 

23 - ‘Do you feel that the flexibility of this class helped you 
succeed in other classes that you were taking in the same quarter?’ 

0.480 

0.020 

13 - ‘How useful were the Quizzes (2,3/4/5,6/7) for providing 
feedback in your understanding of the material for those Modules?’ 

0.463 

0.026 

25 - ‘Compare the quality of interaction between yourself and the 
instructor in this on-line class against what you would have 
expected in a similar ‘face-to-face’ class.’ 

0.439 

0.036 

5 - ‘I feel confident in my ability to access appropriate resources, 
such as subject matter experts, documentation of websites, or late 
breaking news for evaluating uses of space systems in operational 

situations.’ 

0.436 

0.038 

4 - ‘Was the final exam useful in helping you to synthesize the 
concepts learned in the course?’ 

0.434 

0.039 


Table 55. Significant Correlations and Levels of Significance between the Sixteenth 
Question and Other Variables (Ordered by Level of Significance). 


The question about liking website links provided in the course produced very 
favorably distributed responses (Figure 56). The response ‘strongly liked’ dominated 
(twelve students or 52.17%) and the response ‘liked’ followed (eight students or 
34.78%). There were no negative reactions, and only one neutral response. Table 56 

shows nine significant correlations. Responses to questions about the amount of 
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interaction between the students, about liking the module readings, usefulness of module 
readings, class flexibility, etc, are among significantly correlationg variables. 



Figure 56. Distribution of the Responses to the Seventeenth Question. 


N = 23 

Minimum = 4 


Mean = 6.35 
Max = 7 


Median = 7 
Mode = 7 





Statement / Variable 

P 

p value 

26 - ‘Compare the amount of interaction you had with other 
students for this on-line course as compared with a similar resident 
‘face-to-face’ course.’ 

0.633 

0.001 

16 - ‘How did you like the Module readings?’ 

0.632 

0.001 

1 - ‘. .Do you feel that you are able to articulate how physical 
influences on space systems impact our ability to use those systems 
in military operational situations?’ 

0.629 

0.001 

10 - ‘How useful were the on-line Module readings in helping you 
to understand the course concept... ’ 

0.614 

0.001 

23 - ‘Do you feel that the flexibility of this class helped you 
succeed in ofiier classes that you were taking in the same quarter?’ 

0.563 

0.005 

11 - ‘How useful were the linked websites in helping you to 
understand the course concepts, objectives and principles?’ 

0.557 

0.006 

22 - ‘Did you view the flexibility of this on-line course as 
beneficial to your ability to succeed in this class?’ 

0.528 

0.010 

4 - ‘Was the final exam useful in helping you to synthesize the 
concepts learned in the course?’ 

0.453 

0.030 

13 - ‘How useful were the Quizzes (2,3/4/5,6/7) for providing 
feedback in your understanding of the material for those Modules?’ 

0.442 

0.035 


Table 56. Significant Correlations and Levels of Significance between tbe 
Seventeenth Question and Other Variables (Ordered by Level of Significance). 

The eighteenth question was one of five open-ended questions in this instrument. 
It asked: ‘What other instructional materials, if any, should be included in the course?’ 
Five students did not respond. The responses of the others follow: 

‘None* - four opinions 


• Can't think of any 

• N/a for an online course: two responses 

• Nothing that I can see at this time 

• The materials that were provided were adequate 

Suggestions 

• A classroom 

• Better textbook. Classroom discussion. 
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• I think the instructor somehow needs to play a bigger role. The students 
are not the subject matter experts so it got a little old listening to their 
comments. I would prefer to hear from the subject matter e3q)ert. 

• More articles, etc. That's vsdiere I got a better feel for the topics. 

• Perhaps more ‘expert’ participation in the forums. The information 
provided by LCDR Z. was very useful. 

• References to periodicals that the library is on distribution for 

• Search capabilities for forums 

• The Army Handbook should be available either in hardcopy or on CD- 
ROM 


Other notes 

• I thought the textbook was very good, but expensive 


The next open-ended statement asked: ‘Describe your reaction to the instructional 
strategies used to teach the subject matter (textbook readings. Forums, websites)’. Four 
students did not respond. The responses were as follows: 


Positive reactions 


• Excellent 

• Good 

• I approve and have no comments on these strategies 

• I liked it very much. See questions 18 and 21. 

• Learning conceptual ideas were well suited for this particular online 
course 

• No real difficulties 

• The modular readings were great. Short and to the point. You need more 
tests to force one to at least get some understanding out of it. Fewer 
forums. 

• The module readings and websites links greatly simplified the complex 

concepts provided in the text 

• They were all sufficient 

• They were fine. I can't think of a better way to do it... 
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• Too much flexibility 

Neutral remarks 

• Requires self-motivation 

• I didn't read much of the forum discussions due to a lack of time and 
priorities 

• Thought they were good, but honestly don't know how much good the 
quizzes did - i.e. someone could skip the readings and just look up the 
answers. It's really up to the students to do the reading if he wants to learn 
- and this is grad school, so maybe that's the way it should be? 

• Same as question 18. (The response to question 18 was‘None’.) 

Negative reactions and critiques 

• Flexibility was not built into this course. Modules were not completely 
written prior to the course beginning and they were not available until the 
instructor opened them. That reduces the opportunity to work ahead. 

• In a remote environment it would be great. However, at a university it 
does not work for me. 

• Textbook - too technical. Websites - better than textbook. Best part of 
course forums - dull. 

• Textbook was terrible. The forums might have been useful, but after 
awhile it seemed as if we kept talking about the same diings. 


The twentieth question asked: ‘ I am certain that I am able to synthesize 
information firom a variety of sources to enable me to use ^ace systems in operational 
situations.’ As Figure 57 shows, most of the student ‘slightly agreed’ (ten students or 
45.45%) or ‘agreed’ (eight students or 36.36%) with this statement. Only two (9.09%) 
‘slightly disagreed’. Table 57 shows five correlations, which were significant with two 
other statements addressing student abilities to articulate certain subject area issues, 
ability to access appropriate resources, ability to participate in open discussions, and the 
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usefulness of linked websites to help a student understand the course concepts, 
objectives, and principles. 


"I Am Certain that I Am Able to Synthesize 
Information from a Variety of Sources..." 


Number of 
students 





rf®' 


















No response: one 
student 


Figure 57. Distribution of the Responses to the nineteenth Question. 

N = 22 Mean = 5.36 Median = 5 

Minimum = 3 Max = 7 Mode = 5 


Statement / Variable 

P 

p value 

2 — ‘As a result of your experiences in this course, are you able to 
articulate how political, organizational and economic influences 
impact how we use space systems in military operational settings? 

0.634 

0.002 

3 - ‘Do you feel that you are able to articulate how commercial 
space systems will impact military operations?’ 

0.605 

0.002 

5 - ‘I feel confident in my ability to access appropriate resources, 
such as subject matter experts, documentation of websites, or late 
breaking news for evaluating uses of space systems in operational 

situations.’ 

0.509 

0.013 

6 - ‘I feel confident in my ability to get involved in open 
discussions with my peers to weigh issues related to using space 
systems in operational settings.’ 

0.478 

0.021 

11 - ‘How useful were the linked websites in helping you to 
imderstand the course concepts, objectives and principles?’ 

0.475 

0.022 


Table 57. Significant Correlations and Levels of Significance between the Twentieth 
Question and Other Variables (Ordered by Level of Significance). 
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The twenty-first question was also open-ended: “As a result of this on-line course 
experience, do you feel that you have changed your fundamental approach to learning? 
Please respond ‘yss’, ‘someviiat’ or ‘no’, and explain your answer.” Clearly, most of the 
students disagreed they changed their fundamental approach to learning. Below are the 
systematized student responses: 


Yes 

• > Yes, I have learned that there is a great deal of information available 

through many sources and media. The Internet, texts, periodicals, and 
instructor's modules all contributed to a great learning experience. 

• Yes, I plan on researching distant learning for a possible thesis subject 

• Yes, only because I am aware of many of the opportunities provided by 
the NETG program and would feel more comfortable going through some 
of those online courses. 

Somewhat 

• Somewhat no. Still miss the students’ interaction. 

• Somewhat 

• Somewhat. I still think that you can’t fully replace the added value with an 
instructor in a classroom environment. However, I feel that adequate and 
interesting learning can take place on-line. 

• Somewhat. Think on-line is a good venue for certain types of 
courses/curricula. The Space Tech course for ITM is such a combination. 

No 

• No, because I feel that I was already receptive to the distance learning 
concept of education having just completed another 18 months long 
masters program with a different university. 

• No, I need the classroom setting 

• No, I still like to take notes, and have verbal discussion in a short period of 
time, instead of taking an hour(s) explaining/writing a message to post to 
the forum. 

• No, I'm hard headed 
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• No. (Three responses.) 

• No. I did what I had to do to fulfill the requirements of the course. Where I 
was more interested, I put more effort into exploring the topic areas. This 
is how I approach learning in any situation. I don't feel like that has 
changed because of this course. 

• No. I have always searched for additional resources to expand on 
presented material. I like to form unbiased and objective opinions on 
subjective matter. Also, I like to see different explanations on scientific 
matters to ensure vmderstanding. This course reinforced that approach. 

• No. I shall prefer classroom environment with one-to-one approach. 

• No. I still feel I would get more out of in class instruction. Perhaps that is 
just a personal preference. The basic concept here is that you should teach 
yourself via the online class, which is probably ok for something like a 
basic space class. But I would rather have a professor teach me face-to- 
face, allowing for in-class discussion, and focus of concepts and efforts. 

• No. I think this method lends itself to training type courses, not education. 

• No. I thought going in that this would be difficult because I tend to get the 
most out of class discussions vice reading, but wanted to give a try. 1 am 
now more confident than ever that DL is not for me. 

• No... My ways of learning are pretty well set. 

• No... Read and absorb is still primary... 

• Not yet -1 did not put my best forward this quarter, in this class. 


In question 23 of A-2 questionnaire, with question 23 (‘So far, what do you like 
about this on-line course?’), most students responded they liked the time management 
flexibility of the course. The next two questions in the Final survey readdressed that 
issue: one in the context of student success within the course and the another in the sense 
of student success in other classes/courses. Figure 58 shows the distribution of the 
responses to the first question (‘Did you view the flexibility of this on-line course as 
beneficial to your ability to succeed in this class?’). The majority of the students ‘strongly 
agreed’ (eight students or 34.78%) or ‘agreed’ (seven students or 30.43%), but six of 
them (26.09%) disagreed. Table 58 shows the significant correlations. There were seven 


146 



of them: liking the module readings, quality of interaction with the instructor, liking the 
website links, and with three questions dealing with self-perceived abilities or 
confidence. 


"Did You See the Flexibility of this On¬ 
line Course as Beneficial as Your 

Ability to Succeed...?" 

Number e 

of 4 

students 2 
n 
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Figure 58. Distribution of the Responses to the Twenty-Second Question. 


N = 23 Mean = 5.30 Median = 6 

Minimum = 2 Max = 7 Mode = 7 


Statement / Variable 

P 

p value 

16 - ‘How did you like the Module readings?’ 

0.590 

0.003 

1 - ‘. .Do you feel that you are able to articulate how physical 
influences on space systems impact our ability to use those systems 
in military operational situations?’ 

0.585 

0.003 

6 - ‘I feel confident in my ability to get involved in open 
discussions with my peers to weigh issues related to using space 
systems in operational settings.’ 

0.555 

0.006 

25 - ‘Compare the quality of interaction between yomself and the 
instructor in this on-line class against what you would have 
expected in a similar ‘face-to-face’ class.’ 

0.533 

0.009 

17 - ‘How did you like the website links that were provided?’ 

0.528 

0.010 

10 - ‘How useful were the on-line Module readings in helping you 
to imderstand the comse concept... ’ 

0.512 

0.012 

5 - ‘I feel confident in my ability to access appropriate resources, 
such as subject matter experts, documentation of websites, or late 
breaking news for evaluating uses of space systems in operational 

situations.’ 

0.482 

0.020 


Table 58. Significant Correlations and Levels of Significance between the Twenty- 
Second Question and Other Variables (Ordered by Level of Significance). 
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The second question about flexibility had a different distribution of the responses 
than the first one (Figure 59): asked if the flexibility hepled them succeed in other 
clasees, all students agreed with the statement. Ten of them chose ‘agree’ and ten 
‘strongly agree’ (43.48% each). Table 59 shows the correlations with other variables. 
Seven were significant. Only two variables appeared significantly correlated with 
responses to both questions about the helpfulness of the flexibility of the course: liking 
website links and liking module readings. 


"Do You Feel that the Flexibility of this Class Helped 

You Suceed in Other Classes...?" 
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Figiare 59. Distribution of the Responses to the Twenty-Third Question. 


N = 23 

Minimum = 5 


Mean = 6.21 
Max = 7 


Median = 6 
Mode = 7 
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Statement / Variable 

P 

p value 

17 - ‘How did you like the website links that were provided?’ 

0.563 

0.005 

10 - ‘How useful were the on-line Module readings in helping you 
to imderstand the course concept... ’ 

0.536 

0.008 

4 - ‘Was the final exam useful in helping you to synthesize the 
concepts learned in the course?’ 

0.503 

0.014 

26 - ‘Compare the amount of interaction you had with other 
students for this on-line course as compared with a similar resident 
‘face-to-face’ course.’ 

0.500 

0.015 

13 - ‘How useful were the Quizzes (2,3/4/5,6/7) for providing 
feedback in your understanding of the material for those Modules?’ 

0.492 

0.017 

16 - ‘How did you like the Module readings?’ 

0.480 

0.020 

15 - ‘How did you like the textbook reading assignments?’ 

0.425 

0.043 


Table 59. Significant Correlations and Levels of Significance between the Twenty- 
Third Question and Other Variables (Ordered by Level of Significance). 


The twenty-fourth item was the first of foiu dealing with issues of interactivity in 
the course. The first question asked students to: “Compare the amount of interaction you 
had with the instructor for this on-line course as compared with a similar resident ‘face- 
to-face’ course. Note that this question is about ‘amormt’ while another question will 
cover ‘quality of interaction’. Try to consider only the amount of interaction that occurred 
on-line (including e-mails) as opposed to what occurred ifiwhen you stopped by the 
instructor's office.” 

Figure 60 shows the distribution of the responses: most of the students felt there 
were less interaction between the students and the instructor: ten out of twenty-two 
(43.48%) responded ‘significantly less’; five (21.74%) responded ‘less’; and two (8.70%) 
responded ‘slightly less’. Four responses (17.39%) were in the middle of the scale (‘about 
the same’) and two on the opposite end (one [4.35%] responded ‘slightly more’, and one 
[4.35%] responded ‘more’). This variable had significant correlations with only three 
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other variables (Table 60). All three of them actually had some kind of interaction 
between the students and the instructor; quality of interaction between the students and 
the instructor; liking forum assignments (in forums the instructor also interacted with the 
students); and students view on usefulness of the final exam. 


Amount of interaction Between the Student and the 
Instructor in the On-line Course Compared to a Face- 
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Figure 60. Distribution of the Responses to the Twenty-Fourth Statement. 


N = 23 Mean = 2.30 Median = 2 

Minimum = 1 Max = 6 Mode = 1 


Statement / Variable 

P 

p value 

25 - ‘Compare the quality of interaction between yourself and the 
instructor in this on-line class against what you would have 
expected in a similar ‘face-to-face’ class.’ 

0.666 

0.001 

14 — ‘How did you like the Forum assignments?’ 

0.487 


4 - ‘Was the final exam useful in helping you to synthesize the 
concepts learned in the course?’ 

0.456 

0.029 


Table 60. Significant Correlations and Levels of Significance between the Twenty- 
Fourth Question and Other Variables (Ordered by Level of Significance). 
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The next statement about interactivity was the following: “Compare the quality of 
interaction between yourself and the instructor in this on-line class against what you 
would have expected in a similar "face-to-face" class.” 

Figure 61 shows the distribution of the student responses. This distribution is 
different from the distribution of student responses to the question about the amoimt of 
interaction. In this distribution there were six (26.09%) ‘significantly less’ responses and 
eleven (47.83%) ‘about the same’ responses, while in previous question there were 
eleven (47.83%) ‘significantly less ’ responses and four (17.39%) ‘about the same’ 
responses. . 


Table 61 shows significant correlations. 


Quality of Interaction Between the Student and the 
instructor in the On-line Course Compared to a Face- 
to-face Course 
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Figure 61. Distribution of the Responses to the Twenty-Fifth Statement. 

N = 23 Mean = 3.09 Median = 4 


Minimum = 1 Max = 6 Mode = 4 
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Statement / Variable 

P 

p value 

24 - ‘Compare the amoimt of interaction you had with the 
instructor for this on-line course as compared with a similar 
resident ‘face-to-face’ course.’ 

0.666 

0.001 

22 - ‘Did you view the flexibility of this on-line course as 
beneficial to your ability to succeed in this class?’ 

0.533 

0.009 

4 - ‘Was the final exam useful in helping you to synthesize the 
concepts learned in the course?’ 

0.523 

0.011 

16 - ‘How did you like the Module readings?’ 

0.439 

0.036 


Table 61. Significant Correlations and Levels of Significance between the Twenty- 
Fifth Question and Other Variables (Ordered by Level of Significance). 

The twenty-sixth statement, which was the third statement on interactivity, was: 
“Compare the amount of interaction you had with other students for this on-line course 
with a similar resident ‘face-to-face’ course. Note that this question is about ‘amount’ 
while another question will cover ‘quality of interaction’.” 

Figure 62 shows the distribution of responses. The responses seemed to group 
arormd two areas: firstly around ‘significantly less’ and ‘less’ (their total is twelve 
students or 52.17%), and secondly around ‘about the same’ and ‘slightly more’ (nine 
students or 39.13%). Although almost all the responses were present (six of seven 
possible) and student responses were split into two groups with opposite opinions, 
students predominantly thought that amount of the interaction in the course was lower 
than in a face-to-face course. Students significantly differ by perception of the interaction 
when comparing DL to face-to-face course, what is probably significant factor of their 
reactions about the course. However, this finding does not tell us anything about the 
cause of such split in opinions. 
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Amount of interaction Between the Students in the 
On-iine Course Compared to a Face-to-face Course 



Figure 62. Distribution of the Responses to the Twenty-Sixth Statement. 

N = 23 Mean = 2.96 Median = 2 


Minimum = 1 Max = 6 Mode = 2 

This variable had six statistically significant correlations (Table 62), with the 
following variables; liking website links, liking module readings, opinion about the help 
the flexibility of this course provide to succeed in other classes, qiiality of the interaction 
between students, and others. 


Statement / Variable 

P 

p value 

17 — ‘How did you like the website links that were provided?’ 

0.633 

0.001 

16 - ‘How did you like the Module readings?’ 

0.539 

0.008 

27 - ‘Compare the quality of interaction you had with other 
students for this on-line course as compared with a similar resident 
‘face-to-face’ course.’ 

0.511 

0.013 

23 - ‘Do you feel that the flexibility of this class helped you 
succeed in.other classes that you were taking in the same quarter?’ 

0.500 

0.015 

1 - ‘. .Do you feel that you are able to articulate how physical 
influences on space systems impact our ability to use those systems 
in military operational situations?’ 

0.429 

0.041 


Table 62. Significant Correlations and Levels of Significance between the Twenty- 
Sixth Question and Other Variables (Ordered by Level of Significance). 
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The next question asked: ‘Compare the quality of interaction you had with other 
students for this on-line course as compared with a similar resident ’ace-to-face’ course.’ 

The students gave approximately the same number of responses to each category 
from ‘significantly less’ (five or 21.75%) to ‘about the same’ (also five), what makes 
total of eighteen students (78.26%). Fewer students (21.74%) thought the quality was 
better in the on-line class (three students [13.04%]) responded ‘slightly greater quality’, 
one (4.35%) responded ‘greater’, and one ‘significantly greater’). Figure 63 shows the 
distribution. Only two correlations were significant: with the amount of that same 
interaction between the students, and with their opinions on the usefulness of the quiazes 
(Table 63). 
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Figure 63. Distribution of the Responses to the Twenty-Seventh Statement. 

N = 23 Mean = 3.17 Median = 2 

Minimum = 1 Max = 6 Mode = 2 
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Statement / Variable 

P 

p value 

26 - ‘Compare the amount of interaction you had with other 
students for this on-line course as compared with a similar resident 
‘face-to-face’ course.’ 

0.511 

0.013 

13 - ‘How useful were the Quizzes (2,3/4/5,6/7) for providing 
feedback in your understanding of the material for those Modules?’ 

0.492 

0.017 


Table 63. Significant Correlations and Levels of Significance between the Twenty- 
Seventh Question and Other Variables (Ordered by Level of Significance). 

The twenty-eighth question was open-ended. It asked the following: ‘Describe 
your reaction to the media used to present the instruction (i.e. online delivery 
instructional technologies.)’ Three students did not give any response. Below are the 
student responses: 


Positive reactions 

• I felt the media used to present the instruction was quite appropriate. The 
mix of web links and instructor input in the form of the module readings 
made synthesis of the text material much easier and more thorough. 

• I liked them uh-lot 

• Favorable reaction 

• Good when it worked 

• I thought it was fine 

• Interesting 

• It was great when it was working. I am not sure I understand this question 

• It was OK when it worked 

• It works for me! 

• Over all I like it 

• Really enjoyed the links and instructor presentations for modules 

• The media was good when it was up 

• The media was good, with the notable exception of having to pick up 
hardcopies of documents jfrom the instructor. This defeated the purpose of 
online instruction; I should get everything firom the computer. 
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Neutral reactions or suggestions 


• I prefer classroom 

• Need a search engine. Plus better navigational tools 

• No reaction... Just reality 

• None. 

• Thought the media was adequate for the general concepts of the course. I 
thought that if the course were to get more technical, a different media 
would be reqiiired 

Negative or skeptical reactions 

• The technology was not helpful. Caused several delays in the course and 
was not always user friendly. 

• Too much flexibility 

The last open-ended question simply gave students opportunity to add other 
comments. Fifteen of them did not have any additional comments. Those who decided to 
comment responded as follows: 


Positive comments 

• CDR Higgins’ positive energy was a real plus! 

• Great course to teach on-line. The danger here is that because one course 
type works well on-line, we start trying to do other courses on-line that 
just don’t lend themselves well to the venue, like heavy-math type 
courses, EO courses, etc. 

• In this environment I would prefer a classroom environment 


Neutral comments or suggestions 
Need feedback on forum 

The forum should be real-time. Schedule multiple meeting times within 
the groups to have online chats. Then post discussion for bulletin board 
responses from others. 

I do not offer any improvements, I believe a student will get as much out 
of the course as they put into it. The fact that this was a distance learning 
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format did not have a negative impact on my view of the course. CDR 
Higgins always kept students informed and maintained a motivation level 
that is required wiA an on-line course. I will certainly recommend SS3011 
and CDR Higgins to other students. Thanks for all the great instruction! 

• Did you view the flexibility of this on-line course as beneficial to your 
ability to succeed in this class? Yes - when all the technical problems are 
sorted out. However, due to this being for first time the amoimt of 
technical problems removed this flexibility as site was often down when it 
would have been convenient to work on the modules/get the assignment. 
As a result, there were several occasions when we had to cram to complete 
the assignments when it was not a convenient time to work on it because 
the site was down when it would have been convenient. 

• - It's a good course, but not for me this quarter 


The last two categorical but non-ordered variables in the instrument represented 
the on-line version of the Long-Dziuban Reactive Behavior Protocol: a set of four boxes 
Avith responses was used to measure personal on-line behavior types, and another set of 
four boxes with responses was used to measure personal traits. In the first case, the 
students were able to choose only one box, and in the latter case they could choose fi’om 
none, to four traits. Figure 64 shows distribution of the responses on the self-reported 
types. In our sample, the Passive Independent type dominated; there were no responses 
for the Passive Dependent type. 
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Long>Dziuban's Reactive Behavior Protocol - 
Personal Types 
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Figure 64. Distribution of the Responses to Long-Dziuban’s Reactive Behavior 

Protocol - Personal Types. 


This distribution is surprising. Dziuban, Moskal, and Dziuban (1999) found only 
12% of University of Central Florida (UCF) students attending on-line courses (N=381) 
to belong to Passive Independent (PI) type, while in this study eleven students or 50% 
belonged to the PI type. Passive Dependent (PD) was the least frequent type in both 
cases: 5% in Dziuban’s study and none in this thesis research. Aggressive Dependent 
(AD), most frequent in Dziuban’s study (60%), was second most frequent in our sample 
(six students or 27.27%), while 23% of students belonged to Aggressive Independent 
type in Dziuban’s study, similar to this study (five students or 22.73%). 

The sample in this case study is small, and the students were not university 
students as these in Dziuban’s study but military officers selected and self-selected for 
graduate level education in NPS. There is not enough data to conclude whether this 
difference is caused by different (highly competitive) environments, of differences 
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between the two populations, of process of selection of NPS students, or of other factors. 
If Long’s types are interpreted as stabile personality types, the difference would probably 
be the result of the selection and self-selection of students, but because Long and other 
authors indicated that there are possibilities of influence to typical reactive behavior 
patterns (for instance, by counseling them), such a hypothesis is not valid. 

Figures 65 and 66 show responses to the second part of the Protocol, with the 
combined responses shown separated in the first graph, and included in the basic traits in 
the second one. Clearly, compulsive trait, alone (eight students or 42.11%) or combined 
with imp\ilsive trait (five students or 26.32%), dominated. Again, interpretation is 
difficult; it is very hard to distingiush personal preferences, environmental impacts, and 
other possible factors that resulted with this distribution. 



Figure 65. Distribution of the Responses to Long-Dziuban’s Reactive Behavior 
Protocol - Personal Traits, with Separated Combinations. 


According to the authors of the instrument, in this scale a subject can choose any combination of 
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It was difficult to find the appropriate statistical method to check the level of 
association or correlation of responses to this Protocol with other variables in the 
instruments. I decided to assign 1 to the responses a student chose, and 0 to not-chosen 
responses, and I calcvilated correlations with other variables, using only coefficients of 
rank correlation because of a very small number of observations. Table 64 shows all the 
statistically significant correlations. Also, in these calculations I used responses to the 
second part of the Protocol or personal traits with combinations included in basic four 
traits. There were only three significant correlations. First correlation shows that PI type 
was associated with impulsive trait in this sample. Other correlations are very hard to 
interpret. 
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Figure 66. Distribution of the responses to Long-Dziuban’s Reactive Behavior 
Protocol - Personal Traits, with Combinations Included into Basic Traits. 


responses, from none to all four. In this case students chose from none to two traits - which I call 
‘combinations’. 
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Variables 

P 

p value 

‘Passive Independent type’ and ‘Impulsive trait’ 

0.691 

< 0.001 

‘Aggressive Dependent type’ and question 
number 10: ‘How usefUl were the on-line Module 
readings in helping you to understand the course 
concepts, objectives and principles?’ 

0.471 

0.023 

None personal traits chosen and question number 
24: ‘Compare the amovmt of interaction you had with the 

instructor’ 

0.419 

0.047 


Table 64 Significant Correlations and Levels of Significance Between Long- 
Dziuban’s Reactive Behavior Protocol Variables (Personal Traits Include Combinations 

in Basic Traits). 


Final survey - Summary 

The last on-line questionnaire was administered anonymously. Twenty-three of 
thirty students filled out the questionnaire. The questionnaire consisted of thirty-one 
items, including five open-ended questions or statements. 

In terms of student views on their final status in achieving the course lear ning 
objectives, most students replied positively. Most of objectives, according to distributions 
of student responses, were met. Correlations with other variables were significant mostly 
within this same group of variables (asking about meeting different course objectives), 
but there were other significant correlations too. 

The second group of questions addressed how useful the students viewed some of 
the course features: guidelines in the course homepage, guidelines for posting to the 
forums, the textbook, the module on-line readings, linked websites, the forum, exercises, 
and the quizzes as feedback to the students. Most distributions strongly tended to positive 
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responses, but there were significant numbers of negative responses to questions 
addressing usefulness of forums and quizzes. 

The third set of questions asked students how much they liked some of the 
features like forum assignments, textbook reading assignments, the module readings, and 
website links. Responses on usefulness and on liking the same features (module readings, 
forum exercises/assignments) correlated, but coefficients were not high: liking and 
assessing utility of same feature is similar, but is definitely not the same. 

Responses to the open-ended questions showed some trends. Asked what other 
instructional materials should be included in the course, five students responded ‘none’, 
but eight students gave one or more suggestions. Regarding the question about their 
reactions to the instructional strategies used in the course, eleven students responded 
positively, four neutrally, and four responded negatively or with critiques. Asked whether 
this course fundamentally changed their approach to learning, most students (sixteen) 
responded negatively, but three agreed, and four partially agreed. Their reactions about 
on-line delivery of instructional technologies were mostly positive (thirteen responses), 
some neutral (five), and only two negative. For additional comments, the students 
provided three positive points of view, and several neutral comments and suggestions. 

The flexibility of the on-line course in the sense of time management, mostly 
seemed to be more beneficial to the students success in other classes/courses than in this 
course. 

Asked about interactivity, most students thought that the amoimt of interactivity 

between the students and the instructor in the course was less than in a face-to-face 

course, but they were much more imcertain when asked about the quality of that 
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interaction. They were also not sure about the amount of interaction between the students 
and about the quality, but the majority thought that the quality was lower in an on-line 
course. 

The student responses to Long-Dziuban’s Reactive Behavior Protocol were very 
interesting, showing obvious differences from findings in other studies. Because of a 
small number of observations and some other methodological limitations, especially in 
terms of selectivity and self-selectivity that formed this group of students (military 
officers under pressure of competitiveness), an interpretation of these results is difficult. 

B. STUDENT OPINION FORMS (SOFS) 

Twenty-two students completed the Student Opinion Forms anonymously. The 

first part of the SOFs contains data on which curriculum the student attends, about the 

number of credit hours, and about the number of quarters already completed. As already 

stated, all of the students in the sample attended the same curriculum. Figure 67 shows 

the distribution of the numbers of hours per week the students had during the quarter. 


Distribution of Credit Hours in the Sample 
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Figure 67. Weekly Number of Credit Hours Distribution in the Sample. 
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N = 22 


Mean =16.73 
Maximum = 24 


Minimxim = 12 
Standard deviation = 2.90 


Most of the students in the sample had between 14 and credit 19 hours (17 of 22, 
or 77.27%), but the range between the minimum and maximum is relatively large - a 12 
hour difference. The calculation of the rank correlations between the number of hours and 
any other variable in SOFs showed no significant relationships. In other words, credit 
hours did not make any difference in any of the other variables in the instrument. 

The second variable was about the number of quarters a student already 
completed at NFS before the current quarter. Figure 68 shows the distribution of the 
responses with the basic statistical indicators. 
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Figure 68. Distribution of the Number of Quarters Completed. 

N = 22 Mean = 4.00 Minimmn = 0 

Maximum = 6 Standard deviation = 1.63 

The relative majority of the students (10 of 22, or over 45%) were in their sixth 

quarter of their curriculum during the course, but there were individual students with 


164 







none to six quarters finished. Again, there were no significant rank correlations with any 
other variable. 

Answering the next question, all of the students confirmed that the course was 
required in their curriculum and was not an elective. 

After describing basic data, the sixteen statements asked the students for their 
opinions about the course and the instructor. In addition, SOFs provided students an 
opportunity to write their own comments. Responses to the sixteen statements are in the 
form of a five-point scale. For the first eleven statements, the scale had the folllowing 
descriptions: ‘strongly agree’; ‘agree’; ‘no strong opinion’; ‘disagree’; ‘strongly 
disagree’; and ‘no comment’. The latter means the question is irrelevant for the course or 
the student does not want to respond to the question. In the last five statements, where 
students provided an overall rating of the instructor, the course, the textbook, the quality 
of the exams, and the laboratories, responses were: ‘outstanding (among the top 10%)’; 
‘excellent (among the top 30%)’; about average (middle 40%)’; ‘fair (in the lowest 
30%)’; and ‘poor (in the lowest 10%)’. 

I did not include ‘No comment’ or ‘Not applicable’ responses in the graphs. These 
are only mentioned in parentheses because they do not represent an opinion but the 
unwillingness for subjective reasons to give an opinion about that statement. Data on 
median, maximum, minimum, or mode category of the responses to these question do not 
include ‘No comment’ or ‘Not applicable’ category responses. The same holds true for 
rank correlations. For this reason, die higher p value in come cases may be less 
significant because of the smaller number of observations (N) in the calculation. 
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Therefore I ordered the coefficients of rank correlation by significance. If the significance 
was equal, I ordered them by the size of the coefficient. 

The sixteenth statement (‘Overall, I would rate the laboratories’) is not applicable 
for this course and is therefore disregarded. 

The distribution of the responses to the first statement is shown in Figure 69. 


"The Course Was Well Organized" 



Opinion 


Figure 69. Responses to the First Statement in the SOFs. 

N = 22 Mean = 3.59 Median = 4 

Minimum = 1 Maximum = 5 Mode = 4 

From the distribution in Figure 69, student opinions on the organization of the 
course varied firom very negative and moderately negative (three students or 13.64%) or 
neutral (six students or 27.27%), to positive responses, which is the majority (thirteen 
students or 59.09%). In general, most of student were at least neutral or, predominantly, 
positive about course organization. 
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The responses to the first statement (‘The course was well organized’) have 
significant correlations with twelve of the other fourteen variables (Table 65). 


Statement 

number 

Statement 

P 

p value 

7 

‘The instructor was prepared for the course’ 

0.86014 

<0.001 

9 

‘The instructor made this coiurse a worthwhile learning 
experience’ 

0.762 

<0.001 

14 

‘Overall, I would rate the textbook’ 

0.726 

<0.001 

8 

‘The instructor’s objectives for the course have been 
made clear’ 

0.724 

<0.001 

5 

‘I had confidence in the instructor’s knowledge of the 

subject’ 

0.703 

< 0.001 

15 

‘Overall, I would rate the quality of the exams’ 

0.702 

< 0.001 

12 

‘Overall, I would rate the instructor’ 

0.648 

< 0.001 

13 

‘Overall, I would rate this course’ 

0.671 

0.001 

11 

‘The instructor cared about student progress and did 
his share in helping us to learn 

0.654 

0.001 

10 

‘The instructor stimulated my interest in the subject 

area’: 

0.649 

0.001 

6 

‘I felt fi-ee to ask questions’ 

0.644 


4 

‘Difficult concepts were made understandable’ 

0.598 


2 

‘Time in class was spent effectively’ 

0.82515 



Table 65. Correlations and Levels of Significance between Responses to the First 

and to the Other Statements (Ordered by Level of Significance). 

Their responses to the first statement significantly correlated to the responses to 
almost all the other statements; statements that represent desirable qualities of the course, 
the instructor, or the textbook. The organization of the course is very important and is a 
‘related-to-everything’ characteristic within the context of the course. One of the highest 
correlations in this instrument is between the first and the seventh statement. This 


After elimination of ‘No comment’ or ‘Not applicable’ responses, number of pairs of responses (N) 
for calculation of this correlation was 18. N for any o&er variable in this set is 22 minu s the number of ‘No 
comment’ responses, which is visible in the graph of distribution of responses to that variable. 

For the same reason, N of data pairs taken into account was only nine. 
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correlation means that the students’ view on organization of the course and on the quality 
of the instructor’s preparation highly correlates. 

It is interesting to see which question the first question was not significantly 
correlated with. This turned out to be the third question; ‘The instructor seemed to know 
when students didn’t understand the material’. This non-significance of the correlation 
probably appeared because of the changed nature of the feedback from the instructor to 
the students in the DL course. Feedback from the instructor to students in DL is mostly 
individual; therefore, very different than in a ‘normal’ course. Crucial difference in 
comparison with a face-to-face class is that media used for the instructor’s feedback in 
DL covuse (e-mail and other similar means) excludes the instructors’ personal presence. 
Therefore, it was more difficult for students to determine if the instructor was able to see 
student imderstanding of the material. 

The third statement was ‘The instructor seemed to know when students didn’t 
imderstand the material’ Figure 70 shows the following distribution of responses. 

As in many other variables in this set, positive opinions (there are nine such 
responses or 50%) dominate, but there are also seven neutral and one negative opinion. 
Correlations are in Table 66. They are overall less numerous, smaller in size and with 
lower significances than the first statement. 
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"The Instructor Seemed to Know When Students 
didn't Understand the Material" 



Opinion (no opinion: four studente) 


Figure 70. Responses to the Third Statement in the SOFs. 

N=18 Mean = 3.39 Median = 3 

Minimum = 2 Maximum = 5 Mode = 3 


Statement 

number 

Statement 

P 

p value 

11 

‘The instructor cared about student progress and did 
his share in helping us to learn’ 

0.655 

0.006 

4 

‘Difficult concepts were made understandable’ 

0.583 

0.018 

10 

‘The instructor stimulated my interest in the subject 

area. 

0.559 

0.025 

7 

‘The instructor was prepared for the course’ 

0.555 

0.026 

8 

‘The instructor’s objectives for the course have been 
made clear’ 

0.541 

0.031 

9 

‘The instructor made this course a worthwhile learning 
experience’ 

0.537 

0.032 

15 

‘Overall, I would rate the quality of the exams’ 

0.525 

0.037 

6 

‘I felt fi-ee to ask questions’ 

0.486 

0.041 


Table 66. Correlations and Levels of Significance between Responses to the Third 
and to the Other Statements (Ordered by Level of Significance). 


169 



































The next statement in SOFs was the following: ‘Difficult concepts were made 
understandable’. Figure 71 shows the distribution of the responses. Only one student 
disagreed (strongly) with the statement, and one chose not to answer. Again, there were a 
number of neutral opinions (seven or 33.33%), but positive (‘agree’ - nine or 42.86%; 
‘strongly agree’ - fom or 19.05%) dominated. 


"Difficult Concepts Were Made Understandable" 



Opinion (no opinion: 1 student) 


Figure 71. Responses to the Fomth Statement in the SOFs. 

N = 21 Mean = 3.71 Median = 4 

Minimum = 1 Maximum = 5 Mode = 4 

This variable had significant correlations with almost all other variables in the 
instrument (Table 67). The highest correlations were with ‘overall course organizational 
indicators’: overall course rating, how the instructor prepared for the course, and the 
course organization. 
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Statement 

number 

Statement 

P 

p value 

13 

‘Overall, I would rate this course’ 

0.766 

<0.001 

7 

‘The instructor was prepared for the course’ 

0.749 

< 0.001 

1 

‘The course was well organized’ 

0.677 

<0.001 

10 

‘The instructor stimulated my interest in the subject 

area’ 

0.653 

0.001 

6 

‘I felt firee to ask questions’ 

0.689 

0.002 

9 

‘The instructor made this course a worthwhile learning 
experience’ 

0.567 

0.007 

15 

‘Overall, I would rate the quality of the exams’ 

0.578 

0.010 

3 

‘The instructor seemed to know when students didn’t 
understand the material’ 


0.018 

8 

‘The instructor’s objectives for the course have been 
made clear’ 



11 

‘The instructor cared about student progress and did 
his share in helping us to learn’ 

0.504 

0.020 

12 

‘Overall, I would rate the instructor’ 

0.486 

0.030 

5 

‘I had confidence in the instructor’s knowledge of the 
subject’ 

0.481. 

0.032 

14 

‘Overall, I would rate the textbook’ 

0.467 

0.033 


Table 67. Correlations and Levels of Significance between Responses to the Fourth 

and to the Other Statements (Ordered by Level of Significance). 


The fifth question (T had confidence in the instructor’s knowledge of the subject’) 
had almost only positive student responses: fifteen of them (71.43%) chose ‘strongly 
agree’ and five (23.81%) ‘agree’. Only one decided to give a neutral response, and 
another chose not to respond. Figure 72 shows the distribution of the responses, and 
Table 68 lists all the significant correlations with other variables in the instrument. There 
are eleven correlations with a level of significance under 0.05. Interestingly, the highest 
correlation is with responses to the question about feeling firee to ask the instructor 
questions. This is to say, the more students had confidence in the instructor’s knowledge 
and expertise, the more they felt firee to ask her questions within the course context. 
Another high correlation was with responses to the first question about the overall 
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organization of the course. If the students see the course as better organized, the more 
they felt free to ask the instructor questions. 


"i Had Confidence in the Instructor's Knowledge of 

the Subject" 



Opinion (no opinion: 1 student) 


Figure 72. Responses to the Fifth Statement in the SOFs. 


N = 21 

Minimum = 3 


Mean = 4.67 Median = 5 

Maximum = 5 Mode = 5 
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Statement 

number 

Statement 


p value 

6 

‘I felt fi’ee to ask questions’ 

0.800 

< 0.001 

1 

‘The course was well organized’ 

0.702 

<0.001 

11 

‘The instructor cared about student progress and did 
his share in helping us to learn’ 

0.628 

0.002 

8 

‘The instructor’s objectives for the course have been 
made clear’ 

0.606 

0.003 

10 

‘The instructor stimulated my interest in the subject 

area’ 

0.548 

0.010 

9 

‘The instructor made the course a worthwhile learning 
experience’ 

0.535 

0.012 

7 

‘The instructor was prepared for the course’ 

0.550 


12 

‘Overall, I would rate the instructor’ 

0.487 


4 

‘Difficult concepts were made understandable’ 

0.481 


15 

‘Overall, I would rate the quality of the exams’ 

0.459 


14 

‘Overall, I would rate the textbook’ 

0.434 



Table 68. Correlations and Levels of Significance between Responses to the Fifth 
and to the Other Statements (Ordered by Level of Significance). 

I already mentioned the next question (T felt fi'ee to ask questions’) when 
commenting on the previous question. All the responses to this question about feeling 
free to ask the instructor questions during the course were positive — 14 ‘strongly agree’ 
and 8 ‘agree (Figure 73). The issue of the instructor’s very high responsiveness to the 
students was mentioned a number of times in the interviews. Eleven correlations were 
significant, and four correlations had p 0.001 or less (Table 69). 
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"I Felt Free to Ask Questions" 


Number of 
Students 



aJ 

.09''' ^.P» 
























Opinion 


Figure 73. Responses to the Sixth Statement in the SOFs. 

N = 22 Mean = 4.64 Median = 5 

Minimum = 4 Maximum = 5 Mode = 5 


Statement 

number 

Statement 


p value 

5 

T had confidence in the instructor’s knowledge of the 

subject’ 



7 

‘The instructor was prepared for the course’ 

0.729 

<0.001 

8 

‘The instructor’s objectives for the course have been 
made clear’ 

0.719 

<0.001 

1 

‘The course was well organized’ 

0.644 


4 

‘Difficult concepts were made understandable’ 

0.689 

■iKIIllM 

15 

‘Overall, I would rate the quality of the exams’ 

0.659 

0.002 

12 

‘Overall, I would rate the instructor’ 

0.630 

0.002 

11 

‘The instructor cared about student progress and did 
his share in helping us to learn’ 

0.628 

0.002 

10 

‘The instructor stimulated my interest in the subject 

area’ 

0.579 

0.005 

9 

‘The instructor made the course a worthwhile learning 
experience’ 

0.570 

0.006 

3 

‘The instructor seemed to know when students didn’t 
imderstand the material’ 

0.486 

0.041 


Table 69. Correlations and Levels of Significance between Responses to the Sixth 

and to Other Statements (Ordered by Level of Significance). 
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Responses to the sixth question most highly correlated with responses concerning 
the students’ view of the instructor’s knowledge, her preparation for the course, her 
making the objectives clear, and with views on the overall organization of the course. 

The seventh question was about the instructor’s preparedness for the course. 
Figure 74 shows the distribution of the responses. 



Figure 74. Responses to the Seventh Statement in the SOFs. 

N=18 Mean = 4.00 Median = 4 

Minimum = 1 Maximum = 5 Mode = 4 


Only one student (5.56%) responded with disagree (strongly) with the seventh 

statement. Three students (16.67%) chose neutral responses, and four other students 

(22.22%) decided not to respond (‘No comment’). However, correlations (Table 70) are 

numerous and some very high (six of them with p under 0.001). It could be concluded 

that the correlations of student responses to all other variables covered in the SOFs reflect 

very important, and maybe a central role with this issue within the overall context of the 

course. Students clearly think that the instructor’s preparedness has a strong connection 
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with the organization of the course (statement 1), quality of exams (statement 15), 
making difficult concepts understandable (statement 4), freedom to ask questions 
(statement 6), and overall rating of the course (statement 13). 


Statement 

number 

Statement 

P 


1 

‘The course was well organized’ 

0.860 

<0.001 

15 

‘Overall, I would rate the quality of the exams’ 

0.830 


13 

‘Overall, I would rate this course’ 

0.777 

<0.001 

4 

‘Difficult concepts were made imderstandable’ 

0.749 

<0.001 

6 

‘I felt free to ask questions’ 

0.729 

<0.001 

10 

‘The instructor stimulated my interest in the subject 

area’ 

0.715 

< 0.001 

9 

‘The instructor made the course a worthwhile learning 
experience’ 

0.677 

0.002 

8 

‘The instructor’s objectives for the course have been 
made clear’ 



12 

‘Overall, I would rate the instructor’ 

0.632 


14 

‘Overall, I would rate the textbook’ 

0.562 

0.015 

5 

‘I had confidence in the instructor’s knowledge of the 

subject’ 

0.550 

0.018 

3 

‘The instructor seemed to know when students didn’t 
imderstand the material’ 

0.555 

0.026 

11 

‘The instructor cared about student progress and did 
his share in helping us to learn’ 

0.503 

0.033 


Table 70. Correlations and Levels of Significance between Responses to the Seventh 
and to the Other Statements (Ordered by Level of Significance). 

The eighth statement was ‘The instructor’s objectives for the course have been 
made clear’. Although most of students responded with ‘Strongly agree’ (ten of them or 
over 45%), other responses were dispersed over all the other categories (Figure 75). Two 
students (9.09%) strongly disagreed and one disagreed. Four students (18.18%) chose 
neutral responses. In summary, more student responses were positive (total of 15 students 
or 68.18%) than neutral or negative (seven students or less than 32%). 
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"The Instructor's Objectives for the Course Have 
Been Made Clear" 



Opinion 


Figure 75. Responses to the Eighth Statement in the SOFs. 

N = 22 Mean = 3.91 Median = 4 

Minimum = 1 Maximum = 5 Mode = 5 

Interestingly, the variables having the highest correlations (Table 71) with 
responses to this statement (statements niunber 10, 9 and 11) are close by their content to 
the instructor’s approach to the learning process. Through my contacts and interviews 
with the instructor I realized she invested energy to make the course very stimulating to 
students, a worthwhile learning experience, and she cared about the student progress. 
More a student understood the learning objectives of the course, better he or she could 
understand the instructor’s top priorities, what these correlations confirm. 

It would be interesting to examine in depth intercorrelations of these variables, 
but the data (small sample, ordered categorical variables) do not permit more 
sophisticated mathematical-analytical methods for extracting imderlying dimensions and 
relationships. 
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Statement 

P 

p value 

10 

‘The instructor stimulated my interest in the subject 

area’ 

0.810 

<0.001 

9 

‘The instructor made this course a worthwhile learning 
experience’ 

0.752 


11 

‘The instructor cared about student progress and did 
his share in helping us to learn’ 

0.745 

< 0.001 

1 

‘The course was well organized’ 

0.724 

<0.001 

6 

‘I felt fi-ee to ask questions’ 

0.719 

<0.001 

13 

‘Overall, I would rate the course’ 

0.651 

0.001 

12 

‘Overall, I would rate this course’ 

0.656 

0.002 

7 

‘The instructor was prepared for the course’ 

0.662 

0.003 

5 

‘I had confidence in the instructor’s knowledge of the 

subject’ 

0.606 

0.003 

14 

‘Overall, I would rate the quality of the exams’ 

0.569 

0.006 

15 

‘Overall, I would rate the quality of the exams’ 

0.528 

BQIQI 

4 

‘Difficult concepts were made understandable’ 

0.507 


3 

‘The instructor seemed to know when students didn’t 
imderstand the material’ 

0.541 

0.031 


Table 71. Correlations and Levels of Significance between Responses to the Eighth 
and to the Other Statements (Ordered by Level of Significance). 


The ninth statement (‘The instructor made this course a worthwhile learning 
experience’) also produced dispersed student responses, with ‘strongly agree’ responses 
as in the previous question, but with other differences in distribution. Three students 
(13.64%) disagreed with the statement (two ‘strongly disagreed’ and one ‘disagreed’) and 
five (22.73%) chose a neutral response (Figure 76). Significant correlations with 
respective significance levels are in Table 72. There are numerous significant 
correlations, and six of them have p equal or less than 0.001. 
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"The Instructor Made this Course a Worthwhile 
Learning Experience" 



Opinion 


Figure 76. Responses to the Ninth Statement in the SOFs. 

N = 22 Mean = 3.82 Median = 4 


Minimum = 1 Maximum = 5 Mode = 5 


Statement 

number 

Statement 

P 

p value 

12 

‘The instructor cared about student progress and 
did his share in helping us to learn’ 

0.833 

<0.001 

11 

‘The instructor stimulated my interest in the 
subject area’ 

0.816 

<0.001 

1 

‘The course was well organized’ 

0.762 

<0.001 

8 

‘The instructor’s objectives for the course have 
been made clear’ 

0.752 

<0.001 

10 

‘The instructor made the course a worthwhile 
learning experience’ 

0.691 

< 0.001 

13 

‘Overall, I would rate this course’ 

0.672 


7 

‘The instructor was prepared for the course’ 

0.677 


14 

‘Overall, I would rate the textbook’ 

0.600 


6 

‘I felt free to ask questions’ 

0.570 

■ililiM 

5 

‘I had confidence in the instructor’s knowledge of 
the subject’ 

0.535 

0.012 

15 

‘Overall, I would rate the quality of the exams’ 

0.528 

0.016 

3 

‘The instructor seemed to know when students 
didn’t understand the material’ 

0.537 

0.032 


Table 72. Correlations and Levels of Significance between Responses to the Ninth 
and to the Other Statements (Ordered by Level of Significance). 
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The question whether the instructor stimulated students’ interest in the subject 
area was next. Two students (9.09%) disagreed (one strongly), and six students (27.27%) 
chose neutral responses (Figure 77). 


"The Instructor Stimulated My Interest in the Subject 

Area" 


Number of 
students 




Opinion 


Figure 77. Responses to the Tenth Statement in the SOFs. 


N = 22 Mean = 3.95 Median = 4 

Minimum = 1 Maximum = 5 Mode = 5 

All significant correlations with other variables are listed in Table 73. They were 

numerous with seven of them having p equal or less than 0.001. 
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Statement 

number 

Statement 

P 

p value 

8 

‘The instructor’s objectives for the course have been 
made clear’ 

0.810 

<0.001 

13 

‘Overall, I would rate the course’ 



7 

‘The instructor was prepared for the course’ 



9 

‘The instructor made this course a worthwhile learning 
experience’ 



11 

‘The instructor cared about student progress and did 
his share in helping us to learn’ 



4 

‘Difficult concepts were made understandable’ 

0.653 

0.001 

1 

‘The course was well organized’ 

0.649 

0.001 

6 

‘I felt free to ask questions’ 


0.005 

12 ' 

‘Overall, I would rate the instructor’ 


0.007 

5 

‘I had confidence in the instructor’s knowledge of the 
subject’ 

0.548 

0.010 

3 

‘The instructor seemed to know when students didn’t 
understand the material’ 

0.559 

0.025 

15 

‘Overall, I would rate the quality of the exams’ 

0.503 

0.027 

14 

‘Overall, I would rate this textbook’ 

0.471 

0.027 


Table 73. Correlations and Levels of Significance between Responses to the Tenth 
and to the Other Statements (Ordered by Level of Significance) 


It is interesting to note an exceptionally high correlation between the tenth and 
eighth statement. In the students’ opinion, the clearer the instructor made course 
objectives, the more stimulated the students felt. 


The eleventh statement in the Student Opinion Forms was the following: ‘The 
instructor cared about student progress and did his share in helping us to learn’. Figure 78 
shows the distribution of the student responses. Nineteen students (86.36%) agreed with 
the statement, nine of them ‘strongly’. Only three students (13.64%) chose neutral 
responses, and none disagreed. 
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"The Instructor Cared about Student Progress and 

Did His Share in Helping Us to Learn" 

8- 

Number of ® 
students 4. 







0. 



n. 



Opinion 



Figure 78. Responses to the Eleventh Statement in the SOFs. 

N = 22 Mean = 4.36 Median = 4 

Minimum = 3 Maximum = 5 Mode = 4 

Table 74 shows significant correlations with other variables. Eleven correlations 
were significant with p under 0.05, and five had p equal or less than 0.001. The highest 
was the correlation with responses about the instructor making the course a worthwhile 
learning experience. 
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Statement 

number 

Statement 

P 


9 

‘The instructor made the course a worthwhile learning 
experience’ 

0.816 

<0.001 

8 

‘The instructor’s objectives for the course have been 
made clear’ 

0.745 

<0.001 

12 

‘Overall, I would rate the instructor’ 

0.658 

<0.001 

10 

‘The instructor made the course a worthwhile learning 
experience’ 

0.679 

0.001 

1 

‘The course was well organized’ 

0.654 

0001 

5 

‘I had confidence in the instructor’s knowledge of the 

subject’ 

0.628 

0.002 

6 

‘I felt fi'ee to ask questions’ 

0.628 

0.002 

3 

‘The instructor seemed to know when students didn’t 
understand the material’ 

0.655 

0.006 

13 

‘Overall, I would rate the course’ 

0.510 

0.015 

15 

‘Overall, I would rate the textbook’ 

0.481 

0.032 

7 

‘The instructor was prepared for the course’ 

0.503 

0.033 


Table 74. Correlations and Levels of Significance between Responses to the 
Eleventh and to the Other Statements (Ordered by Level of Significance). 

Five five-point rating scale questions followed after the first eleven statements. In 
them the students rated the instructor, the course, the textbook, and the quality of the 
exams. The last scale about the laboratories was disregarded. There were no special 
‘laboratory activities’. 

The first question rated the instructor (Figure 79). Ratings were very positive. 
Seven students (33.33%) rated the instructor as ‘outstanding’ and nine (42.86%) as 
‘excellent’. Only one student (4.76%) chose ‘fair’, one did not rate the instructor, and the 
remaining students (four of them or 19.05%) chose ‘about average’. Looking at the 
correlations (Table 75), we can conclude that in their rating of the instructor, the students 
mostly considered if they thought the course was ‘a worthwhile learning experience’. 
Nine other correlations were significant. 
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"Overall, I Would Rate the Instructor" 


Number of 
Students 



Opinion (no opinion: one student) 


Figure 79. Responses to the Twelfth Statement/Scale in the SOFs. 


N = 21 


Mean = 4.05 Median = 4 


Minimum = 2 


Maximum = 5 Mode = 4 


Statement 

number 

Statement 

P 

p value 

9 

‘The instructor made the course a worthwhile learning 
experience’ 

0.833 

<0.001 

1 

‘The corarse was well organized’ 

0.648 

<0.001 

11 

‘The instructor cared about student progress and did 
his share in helping us to learn’ 

0.658 

0.001 

8 

‘The instructor’s objectives for the course have been 
made clear’ 

0.656 

0.002 

6 

‘I felt free to ask questions’ 

0.630 

0.002 

7 

‘The instructor was prepared for the course’ 

0.632 

0.005 

10 

‘The instructor stimulated my interest in this subject 

area’ 

0.570 

0.007 

13 

‘Overall, I would rate this course’ 



14 

‘Overall, I would rate the textbook; 

0.498 

0.022 

5 

‘I had confidence in the instructor’s knowledge of the 

subject’ 

0.487 

0.025 


Table 75. Correlations and Levels of Significance between Responses to the Twelfth 
and to the Other Statements (Ordered by Level of Significance). 
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The students’ view of the course visibly differs from their view of the instructor. 
The students rated the course less favorably than they rated the instructor (Figure 80). 



Figure 80. Responses to the Thirteenth Statement/Scale in the SOFs. 


N = 22 
Minimum = 1 


Mean = 3.32 
Maximum = 5 


Median = 3 
Mode = 3 


Five students (27.73%) rated the course below average, while only one student 
(4.55%) rated the instructor below average. Fovu students (18.18%) rated the comse as 
‘outstanding’, while seven students (31,82%) chose the same response rating for the 
instructor. 

A possible conclusion from this data is that there are some undesirable elements 
within the course that were not a direct consequence of the instructor’s approach, 
knowledge, teaching or communication skills. An additional and/or alternative 
explanation could be that the student rating of the instructor is more about the instructor’s 


personality, and rating of the course is more about every other factor relevant to the 
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course’s success. However, significant associations among all aspects of the course 
covered with SOFs clearly show that it would be extremely difficult to distinguish the 
subtle factors students take into consideration when assessing course quality. Table 76 
shows significant correlations of the student course rating with other variables. 


Statement 

number 

Statement 

P 

p value 

7 

‘The instructor was prepared for the course’ 

0.777 

< 0.001 

4 

‘Difficult concepts were made imderstandable’ 


<0.001 

1 

‘The course was well organized’ 


<0.001 

9 

‘The instructor made the course a worthwhile learning 
experience’ 

0.672 

0.001 

8 

‘The instructor’s objectives for the course have been 
made clear’ 

0.651 

0.001 

10 

‘The instructor stimulated my interest in this subject 

area’ 

0.633 

0.002 

14 

‘Overall, I would rate the textbook; 

0.565 

Mifjy 

12 

‘Overall, I would rate the instructor’ 

0.564 

■iibikM 

11 

‘The instructor cared about student progress and did 
his share in helping us to learn’ 

0.510 

0.015 

15 

‘Overall, I would rate the quality of the exams’ 

0.525 


6 

‘I felt fi'ee to ask questions’ 

0.512 

■tXIlIM 


Table 76. Correlations and Levels of Significance between Responses to the 

■ Thirteenth and to the Other Statements (Ordered by Level of Significance). 


The fourteenth scale asked the students to rate the textbook. With the SS3011 
course, one main textbook was used. The majority of the students (Figure 81) rated the 
textbook as ‘excellent’ (twelve students or 54.55%) or ‘outstanding’ (six students or 
27.27%), but two students (9.90%) saw the textbook as ‘below average’. There were also 
two (9.09%) ‘about average’ responses. 
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"Overall, I Would Rate the Textbook" 

12 
10 

Number of ® 

Students 4 

2 
0 



Opinion 



Figure 81. Responses to the Fourteenth Statement/Scale in the SOFs. 

N = 22 Mean = 3.95 Median = 4 

Minimum = 1 Maximum = 5 Mode = 4 

Table 77 shows significant correlations with other variables. There were nine 

significant correlations, with the highest correlation with the first statement about the 

organization of the course. 


Statement 

number 

Statement 

P 

p value 

1 

‘The course was well organized’ 

0.726 

<0.001 

9 

‘The instructor made the course a worthwhile learning 
experience’ 

0.600 

0.003 

8 

‘The instructor’s objectives for the course have been 
made clear’ 

0.569 

0.006 

13 

‘Overall, I would rate this course’ 

0.565 

0.006 

7 

‘The instructor was prepared for the course’ 

0.562 

0.015 

12 

‘Overall, I woidd rate the instructor’ 

0.498 

0.022 

10 

‘The instructor stimulated my interest in this subject 

area’ 

0.471 

0.027 

4 

‘Difficult concepts were made imderstandable’ 

0.467 

0.033 

5 

‘I had confidence in the instructor’s knowledge of the 

subject’ 

0.434 

0.049 


Table 77. Correlations and Levels of Significance between Responses to the 

Fourteenth and to the Other Statements (Ordered by Level of Significance). 
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The last scale in this instrument used in this research asked the students about the 
quality of the exams (Figure 82). Responses varied from ‘fair’ (three or 15%) to 
‘outstanding’ (two or 10.00%), with ‘above average’ responses dominating (eleven 
students or 55.00%). Two students did not choose any opinion. 

Eleven correlations were significant (Table 78). The highest correlations were 
with the student’s view of the instructor’s preparedness for the course (seventh statement) 
and with their view of the overall organi 2 ation of the course (first statement). 



Figure 82. Responses to the Fifteenth Statement/Scale in the SOFs. 

N = 20 Mean = 3.50 Median = 4 

Minimum = 2 Maximum = 5 Mode = 4 


188 








Statement 

number 

Statement 

P 

p value 

7 

‘The instructor was prepared for the course’ 

0.830 

<0.001 

1 

‘The course was well organized’ 

0.702 

<0.001 

6 

‘I felt fi«e to ask questions’ 

0.659 

0.002 

4 

‘Difficult concepts were made understandable’ 

0.578 

0.010 

8 

‘The instructor’s objectives for the course have been 
made clear’ 

0.528 

0.014 

9 

‘The instructor made the course a worthwhile learning 
experience’ 

0.529 

0.016 

13 

‘Overall, I would rate this course’ 

0.525 

0.017 

10 

‘The instructor stimulated my interest in this subject 

area’ 

0.503 

0.024 

11 

‘The instructor cared about student progress and did 
his share in helping us to learn’ 

0.481 

0.032 

3 

‘The instructor seemed to know when students didn’t 
understand the material’ 

0.525 

0.037 

5 

‘I had confidence in the instructor’s knowledge of the 
subject’ 

0.459 

0.042 


Table 78. Correlations and Levels of Significance between Responses to the 
Fifteenth and to the Other Statements (Ordered by Level of Significance). 


SOFs also offer students a possibility to add their own comments. Fifteen students 
gave comments. After analysis of their content, I divided the comments to three groups: 

Mostly positive opinions 

• Overall, this was an excellent course. It is crucial however to have backup 
systems. We lost our rhythm when the site went down. 

• Started out as excellent class - web site being down (new o nline course - 
somewhat under standing) greatly hampered tire flexibility of the course -> 
which is supposed to be one of the benefits for an online course. 

• I appreciate the flexibility of the DL format. Cdr Higgins did an excellent 
job in presenting this class in its first on-line format. I learned just as much 
or more than I would have learned in a traditional classroom setting and I 
was afforded the opportunity to do the work as my schedule allowed. If 
given the opportunity, I would choose to enroll in other on-line courses as 
well. 


189 







































Neutral and mixed opinion 


• # 3^6. I have no strong opinion on this question. Instructor’s gage on 
student progress was based on forum input, quiz scores and evaluations. If 
separate contact was made to students doing poorly, I don’t know. 

• This course needs more time to develop if it is to be offered on-line 
exclusively. I am not a proponent of the on-line format as an exclusive 
means of instruction AT a university. It should be xised to alignment 
classroom instruction. I do, however, support its use as a distance learning 
tool. 

• Took the on-line version of course. As an ITM student, I was just looking 
for a god overview of space systems, and that's what the course delivered. 
Anyone looking for MORE than an overview would likely not get what 
they needed in the online version. This is not a commentary on the 
instructor, she was great, more a commentary on not being able to interact 
one-on-one, etc. 

• Constant small group interaction is the key to staying in touch with the 
couree - the more the better. I did not do my part, but this was. a 
worthwhile learning experience. Spread myself too thin this quarter; this 
class did not get the attention that it deserves. 

• First iteration, so I know this, but need to insole no software bugs at UCF. 
Also recommend tests every week. While I appreciated the break, it 
allowed my interaction (especially with readings) to slide. Overall good 
experience. 

• Cdr Higgins knows the material. However there were too many technical 
problems. 

• I feel that the online course has its place within a military setting, i.e. on 
the boat or you are just taking that one class. There is no reason for us to 
take on on-line course while we are here. It was the first class I blew off 
when the other classes got busy. I need the classroom interaction in order 
to learn. I would have rather sat in the classroom and learned something. I 
feel I got nothing out of the online class. 

• If I were to take the course again, I would take the in-class version. I 
enjoyed the material but felt we didn’t get enough conversation in class. 

Negative opinions and critiques 

• Not a good way for me to learn 


It was the statement ‘The instructor seemed to know when the students didn’t understand the 
material’. 
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• Not a good experience at all. Really was a waste of time. Granted it tales a 
lot of effort on both the instructor and student parts. The technical 
problems didn't help, but it seemed as if we were going week to week, you 
can only talk about outsourcing so much. The information covered in 
modules was not enough to properly complete die final point paper. 

• This was a useless class in terms of what we covered. Modules were not 
ready, they were difficult to access 

• Tough 1st go around for online course. Many glitches in the system 
slowed die pace. Need Quizzes on each module just to keep us honest. 

SOFs - Summary 

I made several conclusions and methodological remarks from a summary of the 
analysis of the data from the SOFs. 

The responses offered in the SOFs of ‘No comment’ or ‘Not applicable’ 
decreased the statistical power of the qualitative analysis because such responses had to 
be eliminated from the calculations. 

The number of weekly credit hours (sample average; 16.73, minimum 12, 
maximum 24) and the number of completed quarters (sample average: 4.00, minimum 0, 
maximum 6) do not have any significant correlation to other variables in the set. 

The second statement in the SOFs (‘Time in class was spent effectively’) is not 
applicable in DL environments. It should be revised or removed. 

The third statement in the SOFs (‘The instructor seemed to' know \riien students 
didn’t understand the material’) also seems not to be quite appropriate for a DL 
environment. Its meaning changes for DL courses because of the different nature of the 
feedback from the instructor. 
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In general, most of the responses to the different statements in the SOFs 
significantly correlate. Most significant correlations have statements number one (‘The 
course was well organized’), seven (‘The instructor was prepared for the course’), eight 
(‘The instructor’s objectives for the coiuse have been made clear’), nine (‘The instructor 
made the course a worthwhile learning experience’), and four (‘Difficult concepts were 
made understandable’). Statement number two (‘Time in class was spent effectively’) has 
the lowest number of significant correlations because of the problem stated previously. 

Most of the responses were very favorable about the instructor and the course, 
especially the responses to statement number five (‘I had confidence in the instructor’s 
knowledge of the subject’), six (‘I felt fi'ee to ask questions’), and eleven (‘The instructor 
cared about student progress and did his share in helping us to learn’). To help the reader 
in reviewing all responses in SOFs, I provide summary table of the student responses 
(Table 79) at the end of this summary. 

Although predominantly positive, responses to the majority (11 of 15) of 
statements contained some individual negative responses too. 

Some individual very high correlations provide interesting insights into relations 
within the DL course context. For instance, in the student’s opinion, the more the 
instructor made the course objectives clear, the more their interest in the subject area was 
stimulated. However, the sample size and kind of data available (ordered categories) do 
not allow for more subtle analysis. 
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Frequencies of Responses 


Statements 

(N = 22) 

Strongly disagree 
/ Poor*'^ 

Disagree / Fair 

No strong opinioi 
/ About average 

Agree / Excellent 

Strongly agree / 
Outstanding 

Not applicable / 
No comment 

The course was well organized. 

2 

1 

6 

8 

5 

0 

Time in class was spent effectively. 

0 

0 

5 

3 


13 

The instructor seemed to know when 
students didn’t understand the material. 

0 

3 

B 

6 

2 

B 

Difficult concepts were made 
understandable. 

1 

0 

D 

9 

BB 

I had confidence in the instructor’s 
knowledge of the subject. 

0 

0 

1 

5 

15 

1 

I felt fi-ee to ask questions. 

0 

0 

0 

8 

15 

0 

The instructor was prepared for the 
course. 

1 

0 

3 

8 

6 

B 

The instructor’s objectives for the 
course have been made clear. 

2 

1 

D 

5 

10 

0 

The instructor made this course a 
worthwhile learning experience. 

2 

1 

5 

5 

9 


The instructor stimulated my interest in 
the subject area. 

1 

1 

6 

BBB 

The instructor cared about student 
progress and did her share in helping us 
to learn. 

0 

0 

3 

10 

9 

0 

Overall, I would rate the instructor. 

0 

1 

B 

9 

B 

1 

Overall, I would rate this course. 

2 

3 

n 

6 

D 

0 

Overall, I would rate the textbook. 

1 

1 

2 

12 

6 

0 

Overall, I would rate the quality of the 
exams. 

2 

i 

3 

6 

9 

2 

0 


Table 79. Summaiy of Student Responses in Student Opinion Forms. 


Second responses (‘Poor’, ‘Fair’, etc.) ^ply for last four statements in SOF. 
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C. STUDENT ON-LINE TRACKING 

WebCT software used in the course was able to automatically track the student 

online activities dviring the course. I included only data for the thirty students who 
completed the course in this analysis. Indicators that were interesting for this thesis are: 

• Total number of hits, i.e. number of times a student entered the course web 
pages using his/her logon and password 

• Number of visits to the course homepage 

• Number of visits to the course content page 

• Number of forum articles read, i.e. how many articles in all of forums 
together a student read 

• Number of forum articles posted by a student 

• Number of original forum articles posted 

• Number of follow-up forum articles posted 

• Number of different course pages visited 

The first indicator, total number of hits, was widely distributed, from a minimum 
of 46 to a maximiim of 605 hits by one student. Figure 84 shows the distribution. Most 
students (fifteen or 50.00%) made between 200 and 400 hits during the course, but nin e 
students (30.00%) had over 500 hits. Interestingly, there were two students with less than 
100 hits, but they completed the course also. 

Since the student tracking was recorded by student names, it was possible to 
check not only correlations between the student tracking variables, but also with other 
instruments and variables eissociated with student names, whic)i includes the First 
questionnaire, the Module 14 questionnaire, and final grades. In ceises where both 
variables are numeric and not ordered categories, Pearson’s r correlation coefficient was 
calculated with the respective p value, not Spearman’s coefficient of rank correlations. 
The total number of hits had nine statistically significant correlations (Table 80). 
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Total Number of Hits 


8 
7 
6 

Number of ^ 
students 3 

2 
1 
0 

.jcP -iCP jipP 

6.0'^' 

Number of hits - grouped 


Figure 83. Distribution of the Total Number of Hits. 

N = 30 Mean = 382.93 Minimum = 46 

Maximum = 605 Standard dev. = 149.97 



Instrument 

Variable / Question 

. Pir 

p value 

Student 

tracking 

Number of forum articles read. 

r = 0.711 

< 0.001 

Module 14 

11 - ‘How would you rate your comfort level at 
this point with quiz feature being used to teach 
this course?’ 

p = 0.581 

0.001 

Student 

tracking 

Number of visits to the course content page. 

r = 0.512 

0.004 

Student 

tracking 

Number of visits to the course homepage. 

r = 0.463 

0.010 

Module 14 

10 - ‘How would you rate your comfort level at 
this point with posting comments to the Forums?’ 

p = 0.411 

0.015 

Module 14 

8 - ‘How would you rate your comfort level at 
this point with accessing the Internet?’ 

p = 0.425 

0.019 

Mod\ile 14 

13 - ‘How would you rate your comfort level at 
this point with word processing?’ 

p = 0.404 

0.021 

Student 

tracking 

Number of different pages visited. 

r = 0.395 

0.031 

Module 14 

9 - ‘How would you rate your comfort level at 
this point with sending electronic mail?’ 

p = 0.387 

0.034 


Table 80. Significant Correlations and Levels of Significance between the Total 
Number of Hits and Other Variables (Ordered by Level of Significance). 
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From this table we can see that students who accessed the on-line course materials 


strongly tended to read a larger number of forum articles. Also, they felt more 
comfortable with quizzes, with posting to forums, accessing the Internet, and sending e- 
mail. Also, they tended to visit more different pages in the on-line course material. 

The second student tracking indicator was the number of visits to the course 
homepage. The mean of this variable is much lower than the mean of the total number of 
hits because it was possible to use on-line course materials without visiting the course 
homepage by using favorites or bookmarks in the Internet browser software. Another 
possibility is, in the case of a lost connection to the Internet (e.g., the TCP-IP 
connection), to logon again without reopening the homepage. 

Figme 84 shows the distribution of the visits to the course homepage. There were 
fewer and fewer students as number of homepage visits increase (75-104 visits - eight 
students or 26.67%; 105-134 visits - six students or 20.00%; 135-164 visits - four 
students or 13.33%), but there were also two students (6.67%) with a very low number of 
visits. This indicator had significant correlations with eleven other variables (Table 81): 
six other student tracking indicators, comfort levels with accessing the Internet, posting to 
forums, sending e-mail, and even the number of quarters enrolled. The latter have a 
negative sign, which means that more senior NFS students tended to visit the course 
homepage less, which might mean that they were more able to use course materials 
without starting from the course homepage. There are no clear indicators to support this 
explanation. 
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Number of Visits to the Course Homepage 



Number of visits • grouped 


Figure 84. Distribution of Visits to the Course Homepage. 


N = 30 


Mean = 90.80 Minimum =15 

Maximum =160 Standard dev. = 35.69 


Instrument 

Variable / Question 

p/r 

p value 


Number of original posts to the forums. 

r = 0.619 

<0.001 

Student 

tracking 

Number of visits to the course content page. 

r = 0.553 

0.002 

Student 

tracking 

Number of different pages visited. 

r = 0.514 

0.004 

Student 

tracking 

Number of articles posted to the forums. 

r = 0.508 

0.004 

First 

questionnaire 

Total number of quarters enrolled 

r =-0.479 

0.007 

Module 14 

8 - ‘How would you rate your comfort level at this 
point with accessing the Internet?’ 

p = 0.475 

0.008 

Student 

tracking 

Total number of hits. 

r = 0.463 

0.010 

Student 

tracking 

Number of the forums articles read. 

r = 0.442 

0.015 

Module 14 

6 - ‘Evaluating the likelihood that various types of 
space countermeasures might be used across die 
peace-war continuum.’ 

p = 0.408 

0.025 

Module 14 

10 - ‘How would you rate your comfort level at this 
point with posting comments to the Forums?’ 

p = 0.406 

0.026 

Module 14 

9 - ‘How would you rate your comfort level at this 
point with sending electronic mail?’ 

p = 0.400 

0.029 


Table 81. Significant Correlations and Levels of Significance between the Total 
Number of Visits to the Course Homepage and Other Variables (Ordered by Level of 

Significance). 
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The number of visits to the course content page had almost in shape of a normal 
distribution (Figure 85). This variable had only three significant correlations (Table 82). 
Only correlations with other student tracking indicators were significant. 


Visits to the Course Content Page 


Number of 
students 


12 

10 

8 

6 

4 

2 

0 



Number of visits - grouped 


Figure 85. Distribution of Visits to the Course Content Page. 


N = 30 


Mean = 56.87 Minimum = 16 

Maximxmi = 99 Standard dev. = 21.33 


Instrument 

Variable 

r 

p value 

Student 

tracking 

Number of different pages visited. 

r = 0.716 

< 0.001 

Student 

tracking 

Number of visits to the course homepage. 

r = 0.553 



Total number of hits. 

r = 0.512 

0.004 


Table 82. Significant Correlations and Levels of Significance between the Total 
Number of Different Pages Visited and Other Variables (Ordered by Level of 

Significance). 
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The nximber of forum articles read (accessed) had an extremely wide range - from 
only eight up to 423 forum articles (Figure 87). The total number of articles in the course 
forums was 585, but 34 of them were not available to the students. Only the course 
management staff and the instructor could access them. Also, when the instructor broke 
forum exercises into smaller groups, only members of assigned groups and the instructor 
could access their respective forums. In other words, a maximum of 423 accessed forum 
articles is very close to a theoretical maximum a student could access during the course. 

Eight correlations were significant (Table 83), and they include two other student 
tracking indicators, five comfort levels from the Module 14 questionnaire (word 
processing, quiz, accessing the Internet, sending e-mail, and posting to Forums), and self- 
perceived computer proficiency from the First questionnaire. 

Number of Forum Articles Accessed 

12 

10 

O 

Number of ” 
students ^ 

2 
0 

\CP oJP iSP ^ 

Number of articles read - 
grouped 

Figure 86. Distribution of the Number of Forum Articles Read. 

N = 30 Mean = 239.20 Minimum = 8 

Maximum = 423 Standard dev. = 116.09 
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The next item was the total number of articles posted to the forums. The total 
number can be divided into original posts and follow-up posts, which cause two other 
indicators to be used in this data analysis. Figure 87 shows the distribution of the 
numbers of posted articles. On average, a student posted approximately 14 to 15 articles. 
Most of the students posted between 5 and 20, but the total range is from one to 31 
articles. 


Instrument 

Variable / Question 

p/r 

p value 

Student 

tracking 

Total number of hits. 



Module 14 

13 - ‘How would you rate yom comfort level at 
this point with word processing?’ 

p = 0.478 

0.008 

Module 14 

11 - ‘How would you rate your comfort level at 
this point with quiz feature being used to teach 
this course?’ 

p = 0.466 

0.009 

Module 14 

8 - ‘How would you rate yom comfort level at 
this point with accessing the Internet?’ 

p = 0.461 

0.010 

Student 

tracking 

Number of visits to the course homepage 

r = 0.442 

0.015 

First 

questionnaire 

‘How would you rate your current proficiency in 
using computers’” 



Module 14 

9 - ‘How would you rate your comfort level at 
this point with sending electronic mail?’ 

p = 0.420 

0.021 

Module 14 

10 - ‘How would you rate your comfort level at 
this point with posting comments to the Forums?’ 

p = 0.373 

0.043 


Table 83. Significant Correlations and Levels of Significance between the Total 
Number of Forum Articles Readnnd Other Variables (Ordered by Level of Significance). 

Table 84 shows 10 significant correlations with other variables: the highest 
correlation found in this thesis research is between the number of posted articles and the 
number of follow-up articles posted to the forums. Those students who were sending 
more articles regularly posted more responses to others’ articles in the forums. 
Correlations with comfort levels are also significant with word processing, posting to the 
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forums, and sending e-mail. Two other interesting correlations were with the number of 
quarters enrolled, with senior NFS students posting less articles, and with final grades - 
students more active in posting to the forums tended to get higher final grades. 


Number of Forum Articles Posted 


Number of 
students 



T? 

^ rp' 

Number of articles posted - grouped 


Figure 87. 
N = 30 


Distribution of Forum Articles Posted. 

Mean = 14.53 Minimum = 1 

Maximum = 31 Standard dev. = 7.11 
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Instrument 

Variable / Question 

p/r 

P value 

Student 

tracking 

Number of follow-up posts to the forums. 

r = 0.942 

<0.001 

Student 

tracking 

Number of original posts to the forums. 

r =0.656 

< 0.001 

Student 

tracking 

Number of visits to the course homepage. 



Module 14 

13 - ‘How would you rate your comfort level at 
this point with word processing?’ 

p = 0.501 

0.005 

Module 14 

11 - ‘How would you rate your comfort level at 
this point with quiz feature being used to teach 
this course?’ 

p = 0.497 

0.005 


Number of quarters enrolled 

r =-0.432 

0.017 

Module 14 

10 - ‘How would you rate your comfort level at 
this point with posting comments to the Forums?’ 

p = 0.416 

0.022 

Module 14 

3 - ‘Interacting with a subject-matter expert 
whose knowledge you can use to solve space- 
system problems.’ 

p = 0.404 

0.027 

Module 14 

9 - ‘How would you rate your comfort level at 
this point with sending electronic mail?’ 

p = 0.398 

0.030 


FINAL GRADE 

r = 0.387 

0.035 


Table 84. Significant Correlations and Levels of Significance between the Total 

Number of Forum Articles Posted and Other Variables (Ordered by Level of 

Significance). 

The next indicator compared the number of original posts with follow-up posts. 
Follow-up posts are forum articles written as reactions to posts/articles written by other 
students or the instructor, what is graphically visible in the forums. Figure 88 shows the 
distribution. Most students (twenty-five or 83.33%) posted four to fifteen follow-up 
posts. Three students (10%) posted three such posts or less, and two students (6.67%) 
posted over 20 follow-up posts. 

Table 85 shows significant correlations. There are seven of them. This variable 
correlates with final grades to a higher degree than the overall number of posts to the 
forums. Also, the correlation between the total number of posts and follow-up posts is 


higher than between the total number of posts and original posts. The conclusion derived 
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from these relationships is that the instructor valued originality in the student posting 
forums, what had an impact on student final grades. Three correlations with comfort level 
scales from the Module 14 questionnaires were statistically significant. 


Number of Original Posts to the Forums 



Number of posts - grouped 


Figure 88. Distribution of Original Posts to the Forums. 

N = 30 Mean = 6.13 Minimum =1 


Maximum = 13 Standard dev. = 13.17 


Instrument 

Variable / Question 

p / r 

P value 

Student 

tracking 

Number of articles posted to the forums. 



Student 

tracking 

Number of visits to the course homepage. 




FINAL GRADE 



Module 14 

10 - ‘How would you rate your comfort level at 
this point with posting comments to the Forums?’ 

p = 0.485 

0.007 

Module 14 

11 - ‘How would you rate your comfort level at 
this point with quiz feature being used to teach 
this course?’ 

i 

p = 0.427 

0.019 

Module 14 

13 - ‘How would you rate your comfort level at 
this point with word processing?’ 

p = 0.412 

0.024 

Student 

tracking 

Number of follow-up posts to the forums 

r = 0.378 

0.040 


Table 85. Significant Correlations and Levels of Significance between die Total 
Number of Original Articles Posted to the Forums and Other Variables (Ordered by 

Level of Significance). 
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As already mentioned above, the next variable, the number of follow-up posts to 
the forums (Figure 89) very highly correlates with the total number of forum posts. This 
variable also correlates with the final grades (compare Table 85 to Table 84). 


Number of Follow-up Posts to the Forums 


Number of ® 
students 4 





Number of posts - grouped 


Figure 89. Distribution of Follow-up Posts to the Forums. 


N = 30 


Mean = 9.37 


Minimum = 0 


Maximum = 26 


Standard dev. = 2.14 
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Instrument 

Variable / Question 

p/r 

p value 

Student 

tracking 

Number of follow-up posts to the forums 

r = 0.942 

<0.001 

Module 14 

3 - ‘Interacting with a subject-matter expert 
whose knowledge you can use to solve space- 
system problems.’ 

p = 0.414 

0.023 

Module 14 

11 - ‘How would you rate your comfort level at 
this point with quiz feature being used to teach 
this course?’ 

p = 0.414 

0.024 

Module 14 

13 - ‘How would you rate your comfort level at 
this point with word processing?’ 

p = 0.407 

0.026 

First 

questionnaire 

Number of quarters enrolled. 

r =-0.388 

0.034 


FINAL GRADE 

r = 0.379 

0.039 

Student 

tracking 

Number of original posts to the forums. 

r = 0.378 

0.040 

Module 14 

1 - ‘Identifying key elements of Space Control.’ 

p = 0.367 

0.046 


Table 86. Significant Correlations and Levels of Significance between the Total 
Number of Follow-up Posts to the Forums and Other Variables (Ordered by Level of 

Significance). 


The number of different pages visited was the last of the student tracking 
indicators (Figure 90). Most students (twenty-two or 73.33%) accessed all the available 
pages, but some visited fewer pages: five students (16.67%) visited 14 tol5 pages; one 
(3.33%) visited between 12 and 13, one between 10 and 11, and one only eight web 
pages. There were only four significant correlations (Table 87), but one of them is with 
final grades. Also, self-perceived skills in on-line searching correlate significantly with 
this variable: students with better on-line searching skills found all the relevant course 
pages easier. 
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Number of Different Pages Visited 


Number of 
students 



9 > 




\'j) 

^< 6 ' 


Number of pages - grouped 


Figure 90. Distribution of Numbers of Different Course Pages Visited. 


N = 30 


Mean = 15.8 Minimum = 8 

Maximum =17 Standard dev. = 2.14 


Instrument 

Variable / Question 

p/r 


Student 

tracking 

Number of visits to the course content page. 

r = 0.716 

<0.001 


FINAL GRADE 



Student 

tracking 

Total number of hits. 

r = 0.395 

0.010 

First 

questioimaire 

‘How would you rate your on-line search 
resources and techniques?’ 

p = -0.404 

0.027 


Table 87. Significant Correlations and Levels of Significance between the Total 
Number of Different Number of Pages Visited and Other Variables (Ordered by Level of 

Significance). 


Besides all the listed data from the automated student on-line tracking, I also used 
data on the date and time of all posts in all forums to calculate the number of postings by 
by time during the day, and by day in a week (Figures 91 and 92). This data provides 
additional indicators of students’ on-line activities that could help understanding the 
overall on-line dynamics within the course. 
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Percentage of Forum Posts by Days in Week 



Day 


Figure 91. Distribution of the Posts by Days in the Week (in Percent)^*. 

The regular time to switch to new modules during the course was on Thursdays. 
Figure 91 suggests that the peak of student forum postings were on Thursdays and around 
Thursdays. Thursday was a usual day in a week when the instructor would make a new 
module available to the students and expect all assignment from the previous modiile to 
be completed. Another weekly peak was on Mondays. Students used the convenience of 
the DL course by posting comments on Saturdays and Sundays. 

Figure 92 shows the average number of forums posts by time of day for all 
students and others who participated in the course forums. A 24-hour period is divided 
into 12 two-hours parts. There are three peaks of forum posting: from 9AM to 1PM, from 
3PM to 5PM, and from 9PM to 11PM. By far, the fewest number of posts were sent 
between 1AM and SAM. Rarely did registered course students access the on-line course 

IS Note: this distribution includes all posts to the course forums, what mean not only those by students, 
but by the instructor, technical support staff and others. 
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materials from another time zone. Therefore, this distribution very realistically illustrates 
the circadian^^ aspect of this DL course: NPS students in the sample were active from 
morning to late evening, without indications in this data set that they often take an 
‘afternoon nap’. Moreover, one of peaks of their activity was very late, between 9 and 11 
PM. 



Figure 92. Distribution of the Forum Posts by Time of Day (in percent)^^. 


Student Tracking - Summary 

All eight on-line student tracking indicators appeared to be relevant to this thesis 
They all correlated significantly with between three and eleven other variables from the 
First questionnaire (number of quarters enrolled), the Module 14 questionnaire (comfort 
level in on-line activities and in features used in the course) or with final grades (what 
makes them relevant as secondary indicators of student achievement). With some 


There are numerous studies about circadian nature of human activities, for instance individual 
characteristics related with work in day or ni^t shifts. 

20 Same remark holds for this distribution. 









methodological reserve, all of them have notable correlations with final grades. Analysis 
of their mutual correlations resulted on several conclusions. For instance, the instructor 
valued, in terms of final grades, originality in the student postings to the forums more 
than reactive postings. 

Also, almost all of the student tracking indicators, except two, had between two 
and five significant correlations with comfort level questions in the Module 14 
instrument: comfort with the quiz feature, posting to forums, accessing the Internet, 
sending e-mail, and word processing. 

Additional insight into the distribution of the forum postings by day of week and 
by time of day illustrated the convenience and time flexibility the students had in 
participating in the forums. 


D. FINAL GRADES 

Final grades were also available-for this research. To convert them into numerical 
variables, I assigned points to the grades according to NFS norm: 4.0 points to A, 3.7 
points to A-, 3.3 points to B+, etc. Figure 93 shows the distribution of the final grades. 


209 




Figure 93. Distribution of the Final Grades in the Course (Grades Converted to 
Points: A = 4, A- = 3.7, B+ = 3.3, B = 3.0...). 

N = 30 Mean = 3.71 Minimum = 2.1 

Maximum = 4 Standard dev. = 0.43 

Before listing the significant correlations this important variable had with other 

variables, I have to add an important methodological explanation: some of variables 

correlating significantly with final grades are numerical, so that Pearson’s r coefficient of 

correlation could be calculated. However, because the grades are distributed very 

differently fi'om a normal distribution and because of the small sample size, in these cases 

I also provide p coefficients also. The validity of the coefficients of rank correlation does 

not depend on the distribution or on the size of a sample, so it is advisable to use it in this 

case. Correlations are listed in Table 88. Note, however, that in cases when r or p for a 

numerical variable (student tracking) is not in the table, it means that it is not statistically 

significant (p is greater than 5%). 
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Instrument 

Variable / Question 

p / r 

p value 


Number of different pages visited. 

r = 0.702 

<0.001 

Student 

tracking 

Number or original posts to the forums. 

r = 0.509 

0.004 

p = 0.560 

0.001 

Student 

tracking 

Number of articles posted to the forums. 

r = 0.508 

0.004 

p = 0.591 

0.001 

Student 

tracking 

Number of visits to the course homepage. 

r = 0.507 

0.004 

p = 0.398 

0.030 

Student 

tracking 

Total number of hits. 

p = 0.448 

0.013 

Student 

tracking 

Number of the forum articles read. 

p = 0.441 

0.015 

Module 14 

11 - ‘How would you rate your comfort level at 
this point with quiz feature being used to teach 
this course?’ 

p = 0.465 

0.017 

Student 

tracking 

Number of visits to die course content page. 

r = 0.405 

0.025 


Number of follow-up posts to the forums. 

r = 0.379 

0.039 

p = 0.446 

0.013 

Module 14 

13 - ‘How would you rate your comfort level at 
this point with word processing?’ 

p = 0.395 

0.046 


Table 88. Significant Correlations and Levels of Significance between die Final 
Course Grades and Other Variables (Ordered by Level of Significance). 


Final Grades - Summary 

Final grades significantly correlated with ten variables: with eight online tracking 
indicators (i.e., aU of them used in this research), and with two measures of comfort level 
(with the quiz feature and with the word processing). The correlations do not show 
causality, but they describe relations between sets of collected data. On the other hand, 
this is confirmation that on-line tracking indicators are relevant for the research questions 
in this diesis and can be used as important indicators of several factors in a DL course in 
furdier studies. 
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E. INTERVIEWS WITH THE STUDENTS 


Seventeen out of thirty students agreed to give interviews. 

The instructor thought that only students with very good course grades would 
agree to the interviews. Statistics support that expectation. From seventeen volimteers, 
nine had A and eight A- as a final grade. However, I wouldn’t say that only the best 
students volunteered, but rather that the students with some problems in the course (those 
with the grades below A-) did not volunteer to participate. More than likely, this sample 
of seventeen students is self-selected. 

Interviews were conducted in the quiet part of the Dudley Knox Library at NFS. 
Only the interviewer made notes. At the beginning of each interview, the interviewer 
explained the purpose of the interviews, emphasized that all responses would be used 
only for the research, and stated that the identity of any student would not be revealed in 
the thesis. Most interviews in average lasted between fifteen and twenty minutes. 

In next eleven subsections I present student responses in sequence of the 
questions, grouped by their main meaning. 

1. First Question 

The &st question was: ‘In general, could you estimate have you learned more 
from the on-line designed course than you would learn from a face-to-face SS3011 
course? I'm asking you about the successful learning of the subject matter taught through 
distributed learning, not about a particular instructor or anything else.’ There were not 
any questions similar to this one in the four applied on-line questionnaires or in the SOFs. 

Here are their full responses, grouped into three main types: 
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No 


• Clearly no. Learned lot of basic things, but question-answer dialogue is 
not good. 

• Probably not. Something is missing - interaction. 

• Somewhat less. It was his^i personal fault - lack of motivation to learn. All 
information was available, but he was short on time. It is different when 
one hears something in a classroom - it sounds more serious. 

• Definitely not more. It was hard to read everything. Main text has to be 
read, and then one usually waits for a couple of days and read the 
comments. He had a problem because he likes to have 'duty expert' 
available. The advantages of the DL would be large if he were somewhere 
on duty, not in the school. 

• No, but he got what he needed. It’s matter of student mind set; it’s 
competing. It’s overview of space tech; he got what he wanted. If a course 
were more oriented to his future job, he would give more efforts. 

• Clearly no. He perceives face-to-face instruction as more challenging, 
more demanding, and it is giving more options in what and how to learn. 

• Clearly no. He was exposed to more material, but it was not integrated. 

• Clearly no. It is not possible to throw into DL course all information from 
a face-to-face course and e3q)ect it will function. It all has to be 
reconstructed. There was much information in the course (books, all those 
modules and links). He was missing prioritization, i.e. selection of 
information. 

• He learned less. It’s about self-discipline. Students were not forced to read 
it all, so it was easy losing time. 

• He would probably learn more in face-to-face class. 

• Clearly no. Students did what they had to do. She is personally not very 
interested for this subject area. In a face-to-face course, she would 
probably learn approximately die same or little bit more, what depends on 
instructor. 

Same or undecided 

• Equal, not more or less. Organization was different, easier, but equal 
results. 

• It's hard to measure that. Equally, readings are giving lot of learning. It's 
difficult with materials you dont understand and instructor has to explain - 
this was lacking in the course. 


Le. it was the feult of this student who gave the response. 
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Same. 


• He learned approximately the same; either way offers enough 
opportunities. Maybe less interaction with the instructor, and overview 
class was not very deep, but it was very good - because this course was an 
introduction to the area. 

More 

• She probably learned more from the Internet version. Content was more 
interesting, broader pool of resources. Excellent instructor's job. She 
helped in learning process. It depends on instructor. Interaction was better 
too. She personally likes to express her opinion in calm environment, this 
suited to her. 

• Not in all areas, but in some he learned a lot more, especially about own 
learning style and similar personal learning characteristics. He likes to 
discuss, but he was limiting his posted comments only to few sentences. 
He learned a lot from 3D graphics and from links. He learned very much 
in yisual sense, but not so much in subject matter. 


First Question - Summary 

Overall, their responses may be summarized as: ‘no’ - eleven students; ‘same’ - 
four students; ‘yes’ - two students. Qualitatively analyzing the messages in the ‘no’ 
responses, I found three key elements that students thought caused the on-line comse to 
be less effective than a face-to-face course: 

• The course content and design (explanations: it wasn’t integrated; 
dialogue can’t replace full personal communication; missing selection of 
information; DL course can’t work as face-to-face course). 

• Students and their motivation (personal fault; missing competition or 
challenge). 

• Environment (it would work better from a duty than within the school. 


I found more difficult to interpret student responses in the ‘same’ group. As the 
main justification for their responses, the students reported that though on-line and face- 
to-face courses have a different organization of a learning process (with different 
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amount of interaction), the learning results (effectiveness) are equal. They thought there 
were enough learning opportunities either way. 

Two students responded they learned more from the on-line version of the course. 
One of them was very positive about the on-line course, another only partially. Both 
mentioned terms like ‘learning process’ or ‘learning style’, and emphasized control over 
their interactions with the others, which they considered as an important and desirable 
factor for their opinion about the course. 

2. Second Question 

The second question addressed forums: “Students’ answers about forums were 
very different, as to whether forum exercises were useful in helping students understand 
the subject matter. What is your final opinion about their real usefulness in facilitating 
your learning? Also, comment on how much you liked using forums.” 

Here are the student responses: 


Not useful 

• He didn’t like them. They were too time-consuming: sit, read everything, 
scroll... Some were useful, some less useful, there was too much writing. 
It is easier when the students worked in small groups; otherwise it's too 
diffrcult to control it. Personally he prefers face-to-face discussions, even 
by VTC once a month. 

• He didn’t like forums; didn’t learn much from them. Most of them were 
too focused on some issues, what was not very productive. Instructor 
wasn’t able to moderate the forums well, to encoinage and direct/guide the 
discussions. 

Probably useful 

• Forums seem to be effective, useful interaction tool, but there is a 
drawback - it’s more opinion oriented than factual learning-based. People 
have different background and opinions; it was useful to see that 
difference, but he is not sure how useful that was for learning. If you post 
a comment without reading other posts, all readers can see you were not 
prepared. 
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Useful 


• It would be more useful in more advanced class. Good in sense of 
applying the learned about space systems, but most of material was brand 
new to the students. Interesting, but people often didn’t know what to say, 
tended to go off subject. 

• She liked them. Useful, good references, it's possible to go back to them. 
People normally oppose the change in way they work/fimction as students. 
The course was well organized; contained lot of thoughts. 

• It was useful for learning. He liked forums. Timing was wrong in this type 
of audience, in context of NPS: it should be Monday-Friday. 

• It was more useful in small groups; he liked that. However, the instructor 
was the expert, not the students, so it seemed that real knowledge wasn’t 
‘part of the game’. It was based more as ‘opinion section’, not factually 
based materials. 

• With the given limitations of on-line systems, forums were useful. People 
have different backgrounds and opinions, what he22 likes and thinks ^t 
was very useful. There were some problems here and there; posts should 
be limited by length. 

• Yes, useful, he got majority of what he gained it the course from reading 
forum messages, even though it was not much a direct learning. He 
enjoyed them, liked them. He learned mostly from texts. 

• Beneficial, even more after dividing the class into smaller groups. It takes 
time, but this is closest to classroom interaction you can get. He liked 
forums. 

• Useful. He recommended division of forums into smaller groups, what the 
instructor did later. Many students posted because of grades; they knew it 
was expected of them. Some had excellent ideas, but they wrote too much. 
Instructor should be more specific about topics of die forums and should 
limit the length of posts. Many thoughts are being repeated. Maybe forums 
should be made only for some of the modules. 

• He liked them - he heard different opinions, everybody thought 
differently. When they were easy for use and functional - it was very 
good. When there were technical problems with posting to forums, there 
was no use of them. 

• Yes, they were useful, one of the highlights of the course. He liked them. 
Some people were very active in forums, have written a lot. 


22 He - the interviewed student 
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• She liked them. They were useful. They forced students not to just react, 
but to form an opinion and to ask for more explanations. Some people are 
not comfortable speaking in class, but in forums they gave more, they 
participated more. Forums helped to get together more different opinions 
and to explore different opinions. It was different than class environment. 

• Yes, useful, but he didn't like them. Some students had excess time - they 
wrote very much. He wrote brief texts. He didn’t like lack of 
responsiveness in the forums. 

Very useful 

• She liked them. Open dialogue, and people more forthcoming with their 
opinions. She would prefer chat (real time) as addition. Very useful, 
especially towards the end. All in all, very well roimded opinion. 

• He saw forums as very useful; lot of reading involved, challenging 
because one has to be careful what to write. Weakness is that you should 
read all previous postings. There was no way to search all posted 
information; it would be a very useful capability, way to cataloguing 
posts. Personally, he works a lot on development of web applications. 


Second Question - Summary 

If we reduce the student responses to this question in the interviews to key 
messages/responses, tiiey would be: ‘not useful’ - two responses; ‘probably useful’ - one 
response; useful — 12 responses; and ‘very useful’ — two responses. Analysis of the 
responses suggests that the students used the same argument about student opinions to 
support quite opposite responses. Here are the factors they emphasized for their responses 
to this question: 


Responses ‘not useful’ 

• Not much to learn because posts are opinion-driven, too focused, not 
factual’. 

• Too time consriming. 

Responses ‘useful’ 

• Diverse, opinion-driven and background-diverse posts. (Five responses.) 
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• Post can serve as good references. 

• Forums acted as replacement for class interaction. 

• Forums help some students, otherwise very quiet, to express more freely. 

• Posts more discuss applications than theoretical aspects of the space 
systems issues. 

Response ^probably useful* 

• Dilemma (or drawback): forums may be useful or not because they are 
opinion-driven. 

Responses *very useful* 

• Students are forthcoming with their opinions - open dialogue. 


3. Third Question 

The third question addressed quizzes: ‘Same question about Quizzes - please 
comment on how useful they were for you and how much you liked them!’ 


Here are the responses to the third question: 

Not useful 

• Not useful. They didn’t have enough depth. It was possible to answer by 
reading from the textbook. He didn’t like them. 

• Absolutely not useful. Too easy, they^^ could do five runs, with the 
book.24 They did not demand students to think. She didn’t mind doing 
them. Maybe some essay-type questions could be added, especially in 
second half of the course. 

Not very useful 

• Not very useful. They were never completely fixed. They were easy, but 
one had to read the book. They should be more challenging. 


23 The students. 

2^ Quizzes provided students with feedback about incorrectly answered questions, and enabled them to 
go tbrough all questions iq) to five times. Because quizzes were on-line, it was possible for students to tise 
the textbook and notes \^le answering quiz questions. 
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• Not very useful. It was more like a test have you read the material at all, 
than anything else. It would be better if quizzes were directed towards 
some knowledge test at the end of the course. 

• If open book and more dian one pass, not much use. 

• They seemed pointless; they didn’t capture students’ understanding of 
subject. 

• It was just a drill, not very useful: open book, no restrictions, almost 
anyone with the book could do to them. 

• He liked them in sense of not demanding too much, but the facts from 
quizzes didn’t remain in knowledge; in essence it had not much use. 

Useful 

• Good as a review process. Open-book so student could choose not to 
really leam about topics from quizzes but simply to read from the book. 
Problems with grading were very frustrating. It was too easy, but good 
subject matter issues were present in content of the quizzes. The quizzes 
were too simple. 

• Yes, useful, forcing you to read. That was the point. Easy. 

• They were not difficult; he liked them because they contained information 
and facts, not opinions. 

• Yes, useful, forcing students to do the reading. For DL (web), self- 
discipline is very important. Quizzes were pretty relevant. Otherwise it 
would be difficult for a student to identify where he stands, what he has to 
do. They were definitely a roadmap of the course, important feedback. 

• Yes, they were useful, but not fair in sense that they can be answered in 
collaboration with other students. They should be more challenging, but 
they were not too easy. He neither likes nor dislikes quizzes. 

• Useful, they let students know what they should have learned. Make 
students to HAVE to read/leam material. He didn't like them much; he's 
neutral. To have several runs in a quiz is counterproductive. Quizzes didn't 
demand true understanding from the students. 

• Quizzes were appropriate. They help to focus study, help to understand 
what's important, what you needed to leam. They do not need to be 
difficult to do that. He liked them. 

• They were too easy. There were some technical problems at the beginning. 
It is good that more runs are possible in quizzes, but then their usefulness 
(and fairness) becomes questionable. The quizzes should be a little deeper, 
more comprehensive. Maybe only two tries, not five. Also, maybe few 
essay-type questions should be added. 
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• They were tied closely to the book, without them students wouldn’t read 
as much. Quizzes were good, forces you to learn but not too hard. Helpful 
- forced students to self-organize. 

• He liked quizzes and there should be more of them; otherwise he would 
study too much in last minute. He liked the possibility of redoing the quiz. 
They are useful for the learning process. Bugs made some of the students 
angry. Quizzes were ‘confidence builders’; they were balanced mixture of 
course content, some maybe even too condensed. 

Third Question - Summary 

Summarized, student responses were: ‘not useful’ - two responses; ‘not very 
useful’ - four responses; unclear - one response; ‘useful’ - ten responses. After analyzing 
these responses, I concluded that the key reason for such opposite views about the 
quizzes was different interpretation of the role of the quizzes. For instance, both 
groups of students who saw the quizzes as either not useful or as useful mentioned that 
they were easy. However, they saw role of quizzes in the course completely differently. 
Also, some students thought the quizzes were easy but did not like them. These 
students were used to competitive, demanding, and challenging tests, quizzes, and exams. 
Consequently, these quizzes (with five possible tries and with no time or closed-book 
constraints) were not challenging them. In fact, quizzes that are too easy may seem 
offensive to them. 

4. Fourth Question 

The fourth through seventh question covered issues of interactivity. The 
statements asked the students to comment the amount or quality of the interaction 
between the students and the instructor or between the students. 

The fourth question was: “Students’ answers to the question about how they saw 
the amount of interaction with the instructor during the SS3011 course, compared to a 
similar resident (face-to-face) course, were very different. Please provide your view.” 
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are the responses to fourth question; 


Much less 

Much less interaction. She felt depersonalization. The instructor didn't 
know some of the students. Not much one-to-one interaction. 

A lot was lost. No body language. It was time consuming, but die students 
lost important part of communication. 

Much less interaction in on-line than students needed. The instructor 
worked hard ‘behind the curtain’. At the be ginning the instructor posted 
many comments, but later less - it was lacking. 

Much less. (Two responses). 

Much less amount. Personally he likes to talk a lot, to personally ask the 
instructor. His interaction was clearly less. 


Less 

Not enough, maybe the instructor wanted to be ‘invisible’ too much. 

It was less, but it wasn’t so bad. It’s simply different environment (for the 
students). 

Significantly less. 

Less. There is no personal interaction like in a classroom, but the 
instructor can better focus the interaction he/she has, and to use it more 
efficiently. 

The problem was that DL course was organized in local environment. If 
students had to take written materials firom the instructor, then DL looses 
its sense. The instructor was giving her best, responded to all e-mail 
messages, talked a lot, and that was very good. The amount (of interaction 
of students witii the instructor) was a little less. 

Less. 

Less. Most of interaction in forums was about technical problems. The 
iiistructor pretty often participated in the forums, mostly in form of 
response to one person's post. 

Not enough. 


Unclear 


• It is an individual interaction with the instructor. In face-to-face courses, 
students rarely exchange e-mail with instructors. That was the good side of 
on-line version of this course. 


Enough 

• There was enough interaction; it was possible to contact the instructor 
anytime, or other students. 

More 

• In fact, there was MORE interaction. He had another on-line course, very 
structured, without almost any interaction with the instructor. In SS3011, 
the instructor was very dynamic, stimulative, responsive, and she was key 
factor for the success of the course. 


Fourth Question - Summary 

Reponses may be summarized as following: ‘much less’ - six comments; ‘less’ - 
eight comments; unclear - one comment; ‘enough’ - one comment; ‘more’ - one 
comment. Analyzing the ‘much less’ responses, I concluded that those students felt 
uncomfortable because of the absence of the instructor’s personal presence. They 
mentioned terms like ‘body language’, ‘depersonalization’, and similar terms without any 
other clear justifications for such responses. The student who responded ‘more’ in fact 
was comparing this course to another on-line course, but with much less interactivity. 
Other students gave a wide spectrum of explanations. It seems that their approach to DL 
interactivity was more diverse than approach of the students who responded ‘much less’. 
5. Fifth Question 

The next statement was: ‘I want to ask you similar question about the quality of 
students’ interaction with the instructor, compared to a resident course. How did you 
see it?’ 
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Here are full responses of the students to fifth question: 

Less 

• It decreases if it’s only all in words. With multimedia and similar 
improvements, it’s better, but there are more lost aspects. Very time 
consuming. It’s much better in small groups, more personable. 

• Usually someone’s opinion can be understood better if you’re facing that 
person. Otherwise, that is lost. The instructor wasn’t present enough. 
She25 ‘intervened more after the fact’ than she was engaged. 

• It depends on instructor. If the instructor is active, reactive, responsive, 
then it's OK. Maybe little bit lower quality than in face-to-face course. 

• Clearly less effective. Instructor was relatively OK, trying to be involved 
as much as possible. IW^^ class - the students barely heard firom the 
instructor at all, that was hands-off process. 

• Nothing can really replace face-to-face, even though the instructor was 
very responsive. He felt the need to talk to her. 

Unclear 

• The instructor was very good; she was able to answer every question, what 
increases quality. However, she didn’t force all students to enter into 
interaction with her. Deeper integration is missing, nonverbal 
communication not present. 

• That’s tough to achieve. The instructor was very responsive, but maybe 
not enough for him personally. He felt something was missing to conclude 
each of the modules, maybe real-time on-line (chat) session about it, or at 
least the instructor sending some kind of conclusion to each student (some 
‘walk-away’ points). 


Same 

Probably the same as in face-to-face. The students had possibility to visit 
personally or contact by phone the instructor, so they were able to increase 
quantity and quality of this interaction if they needed to. 

Good 


The instructor. 

26 Information warfere. 
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• The class was too large. Instructor didn’t have enough time. Quality was 
good. 

• Good quality. The instructor’s comments were good, relevant. 

• It was better than in face-to-face; it was individual feedback. 

• Interaction with the instructor was mostly by e-mail, but it was good. 
Whenever he personally met instructor, interaction immediately started. 
There were no occasions when the instructor didn't respond. 

• The quality was pretty good. However, responses from the instructor 
should be quicker. 

• It was fine; the instructor was sending comments throughout, but it’s 
limited. 

• It was good. The instructor was responsive, though not quite enough. For 
him personally, face-to-face is ‘better quality’ because of his learning 
style. In face-to-face class, stupid question is quickly forgotten, but in DL 
it stays on server. In terms of contacting with people from space systems 
practice, nothing is good as DL. 

Better 

• The quality is better. There is always an option to initiate more interaction 
if you need it. There is a potential here for much more interaction (like in 
face-to-face), but it's not really as necessary. 

• Not diminished, maybe even better. She^^ had problems with the password 
(SSN). The instructor was doing her best. She was veiy responsive. 


Fifth Question - Summary 

Summarized, the responses were: ‘less’ — five responses; unclear — three 
responses; ‘good’ - seven responses; ‘better’ - two responses. Explaining the ‘less’ 
responses (there were five such opinions), students emphasized the lack of the 
instructor’s personal presence, which was also similar to the explanations for the ‘much 
less’ responses. Ten students thought that the quality of the interaction was good, the 
same, or even better than in the face-to-face class. Even though many of these ten 
students mentioned other problems such as too large a class, problems with passwords. 


27 The student. 
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tile instructor’s responses was not quick enough, personal preference with face-to-face 
classes, they still saw tiie quality of the interaction better anyway. 

6. Sixth Question 

The sixth question asked: ‘One of most controversial questions, according to the 
students’ answers, was about the amount of interaction between the students, 
compared to a face-to-face course. Some thought SS3011 offered significantly less 
interaction, some others - significantly more. What’s your comment?’ 


The complete responses to sixth question were: 

Much less 

• Much less interaction than in face-to-face courses. Majority of interaction 
was between the students. Most of interaction about the course in the on¬ 
line mode was about the course itself, not about topics. 

• Much less - he met almost nobody, only one 

• Very low amount 

Less 

• For majority of students it was purely on-line. Amount is limited. 

• Significantly less than in face-to-face course. He was thinking a lot about 
applicability of this knowledge, he was expecting a ’toolbox’ firom the 
course. Students talked about that a lot. 

• Less than in normal class. Tempo was fi'agmented; many were working 
only once a week. There were many postings; it was expected firom the 
students. The quantity was not at the same level as quality. 

• Forums were there for this interaction. Of course it was different than in a 
classroom (face-to-face). Less interaction. 

• Besides forums, not much interaction. DL form decreased interaction. 

Little less 

• Probably little less because there was less informal interaction which 
usually goes in a classroom. He didn’t have any informal communication. 
Somevdiat less than in face-to-face course. 
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• Not so bad. He felt good and bad sides that; maybe bad sides a little bit 
more. If he were on duty, on a ship or something like that, it would be 
OK! Here in school occasionally it was frustrating. It even happened that 
students were sitting one next to the other and they were exchanging e- 
mail! 

• Maybe the same or little bit less. If students sit together in a classroom, of 
course they exchange more information. 

Same and unclear 

• Purely within the forums (at least for her). Pretty good interaction. 
Everybody can respond to everybody. On-line delivery increased quality 
and quantity of interaction between the students. 

• About the same. Students in NPS talked at other classes too. Mostly, it 
was OK. 

More 

• The difference in interaction was almost unfair. There was a lot of 
interaction, because the students were forced to respond to posts. 

• There was more interaction here; in face-to-face courses every student is 
concentrated on the instructor. SS3011 students were with different 
backgrounds, but exposed to the same material and they became closer. 
Maybe the instructor didn’t mediate enough; she ought to direct more 
attention towards the key points. 

• More, because it was required to post responses and thoughts. In usual 
classroom there are always some students A\iio do not talk much. Here all 
students were forced to participate, so there is more interaction than in 
face-to-face class. Good quality and quantity of interaction between the 
students, but for a high price (again, he personally prefers ‘face-to-face’). 
He simply doesn’t like to write. 


Sixth Question - Summary 

I grouped student responses by their key messages as follows: ‘much less’ - three 

responses; ‘less’ - five responses; ‘little less’ - three responses; ‘same’ or unclear - two 

responses; ‘more’ - three responses. Interestingly, there was a broad spectrum of 

different answers to this question. Most students (eleven out of seventeen), thought that 

there was interaction between the DL students compared to a face-to-face course. It is 
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very difficult to interpret this data, but it seems that these student subgroups with 
different opinion might have a different focus in face-to-face classes: one focused their 
attention more on the course instructor, and the other more on the teaching material in the 
course. If so, their perception of the change of interactivity in the DL environment might 
be so opposite, as it was in the example of the responses to this statement. 

7. Seventh Question 

The next question asked: ‘The last question about interaction covers the issue of 
quality of interaction between the students, what is important for the learning process.^* 
How would you comment it, in sense of comparing the quality of interaction between 
students in on-line SS3011 with interaction between students in some similar face-to- 
face course?’ 


Here is how the students responded to seventh question: 

Worse 

• Worse, same answer as in question 6. Synergy never develops. 

• The quality was suffering, but it wasn't because of the instructor but 
because of media used 0L). 

• The interaction quality was not as good as in a face-to-face course. 
Limited, time-constrained. No real-time interaction. The best sort of 
learning interactivity is arguing in class. 

Same or unclear 

• Fair, in die middle of the scale (2.5/5). Before the end of the course he 
understood the instructor's expectations/demands, but that cant be done 
mechanically, a dialogue is necessary. When one starts to feel that he isn’t 
getting anything more from new postings, then there is no more need to be 
engaged in that dialogue. At one moment, it isn’t stimulative any more. 
Time factor is very important 


Research and experiences with DL at UCF and the instructor’s and student statements during this 
course agreed about die importance of interactivity for efficient learning in a DL course. 
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It depends: some students thought well before posting an answer. Others 
went directly with their opinion, without many facts in the content. 

Forums were best part of interactivity in this course. They reflect that 
quality of interaction. Some students were expressing own opinions; 
others were reacting, or were not able to react if everything is already said. 

Some people have more difficulty talking in front of the others. Also, if 
on-line, one has to be careful what he is writing. It would be better if an 
instructor told to students to feel free to write their own opinion (with a 
level of privacy too!), and it would not be sent to anybody else, but she 
can’t remember that the instructor in this course told anything like that. 

It was laborious. He would like more real-time discussion. There was 
quality, but maybe not in right format. 

Little better 

Probably little higher quality, because one has to think well before posting 
message/ comment. 


Better 

Pretty good quality. Casual interaction from face-to-face courses was 
lacking here. 

Pretty good 

Quality was better. One has to research what he wants to say - can’t write 
too much. He personally used to revise his posts before sending. 

Academically, the quality was better. It was also good that posts remain as 
references. So one has to be careful what to post, have to check it in the 
book. 

It was hard, but probably better because of much written communication. 

Yes, better quality, but different kind of interaction. Personally he prefers 
not to write, so he was very careful in writing - that was a challenge for 
him. 

Probably higher quality. The instructor never directly commented on 
postings. 

He agrees that one should think before posting (it stays on server), but 
there are some students who posted (in forums) only because they had to. 
Quality was higher. 



Seventh Question - Summary 

Smmnarized, the student responses were: ‘worse’ — three responses; ‘same’ or 
unclear - five responses; ‘little better’ - one req)onse; ‘better’ - eight responses. Clearly, 
when the question is about the quality of the interaction between the students in the DL 
environment, the students have significantly more positive than negative opinions, 
compared to the question on the amount of the same type of interaction. Nine students 
thought the quality of the interaction was better in DL than in a face-to-face course, and 
only three students thought the opposite. Several students stated diat one of the key 
reasons for this opinion was the fact that posted forum articles are written material that 
stays on the server as a reference. Therefore, it is necessary to be careful about what is 
posted. 


8. Eighth Question 

The eighth question in the interviews was: ‘If you have an opportunity to choose 
between on-line and face-to-face section of some other class in your curriculum, what 
would you do? (Examples: I would sxirely go online; depends on the nature of the course; 
depends oh my schedule; surely stay in face-to-face section...)’ 

Their responses to eighth question, grouped, were: 

Face-to-face 

• While in NPS, preferably face-to-face, but not 100%. In some situation 
DL gives far better flexibility, but the quality is important issue too. Face- 
to-face has some big advantages. If he is in some degree education 
program, it depends; he would probably take it 100% on-line. 

• Face to face, mostly. He has a face-to-face course that is perfect for on¬ 
line delivering. 

• Clearly face-to-face, it fits better his learning style. He gains a lot fi'om 
interactions than firom quantity. 

• In the school he would take face-to-face, but if out of the school, or taking 
only some single external course, he’d take on-line. 
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In NPS - face-to-face. If on duty, of course or traveling who knows where 
- he would take on-line. 


Depends 

It depends on type of a course. History or similar (one-way dialogue) - on¬ 
line is great. Math or similar courses demand interaction, so he would 
prefer face-to-face. 

Depends on the course. More complex, dynamic courses would be very 
hard on-line. More basic, core courses can be on-line. She had Information 
Warfare course on-line; they were conceptualized differently. 

It depends on the course. If something from history or political sciences, 
surely on-line. Math: face-to-face. SS3011 is somewhere in the middle. It 
suffered a little as on-line, and it depends much on depth of solving 
technical problems and similar issues. The course was more at a general 
level. 

Depends on the course. Math - face-to-face. This course - OK for on-line. 

It depends on: 1. Course. 2. Instructor (what he knows about instructor). 3. 
Time available. Example: now he has a course, which is all in Power Point 
format. Such course should be on-line, because course instructor is not 
necessary. SS3011: students and the instructor should meet at least once a 
week, even using VTC. 

Depends on the content. Some courses like history or social sciences are 
better on-line. Math and similar - better face-to-face. 

If course is similar to this one, he would take it on-line, because it gives a 
lot of flexibility. This is the main advantage. If something is more 
difficult, like math, he’d take it face-to-face. 

It depends on course material. More technical courses (math, engineering) 
- he would choose traditional classroom, because he is not as experienced 
in that area. He needs advantages of a face-to-face class. 

Depends on type of a course and on the environment (in the school or off 
school). In the school he would mostly choose a facC-to-face section. If on 
job, he would clearly choose on-line, because of flexibility. 

Depends on instructor. Short courses (compressed into three days or so) 
should be taught via DL, but if three weeks or more - face-to-face. After 
some time interest decreases. If a course is not demanding, attention and 
interest diminish. 

Probably face-to-face, but it depends on the course. If something is not 
interesting to him, DL wouldn’t help. 
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• It depends on the nature of a course. If it’s required for a degree, but she is 
not particularly interested in that course, she would take it on-line. If it’s 
one of more interrelated courses (math or similar), she would take it face- 
to-face, especially if she doesn’t have background in that area. In any case, 
it’s GOOD option! 

Eighth Question - Summary 

Of the seventeen students interviewed, five responded ‘face-to-face’ and twelve 
responded ‘depends’. Only five students said they preferred the ‘face-to-face’ version of 
the courses, but even within that group, four students added that they would change their 
opinion if they were not at NPS. 


The students who responded ‘depends’ gave three typical explanations of what 
‘depends’ meant to them: on the environment (in a school or firom duly), on the nature of 
a course (if a less mathematical or technical course - the student woiild more likely take 
the on-line version), and on an instructor. 

9. Ninth Question 

The ninth interview item asked students: ‘Please tell us any 
advice/recommendation/suggestion you have for CDR Higgins or any other NPS or 
Navy distributed learning instructor.’ 

• Have to have face-to-face meetings every 14 days or so, if possible. Tech 
things should be fixed before the start of the course - studrats felt like 
subject of experiment. 

• The instructor was very responsive, and that was good. Instructor has to be 
very involved, has to guide. 

• He liked forums. However, a student has to be careful with time dedicated 
to the forums because of other courses. Quizes shouldn’t be too easy it 
they’re on-line. Face-to-face quizzes are twice as difficult. There were 
some tech problems at the beginning. The instructor’s feedback has to be 
sent to individual students or groups. CDR Higging made some 
observations about whole class, but nothing more than that. 

• There must a backup plan, even a second backup for the course too. 
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• Advantage of DL is time (whenever you choose). So, entire course has to 
be ready and available in advance (or least, large parts of the course). 

• On-line courses have to be non math-intensive or similar types of classes 
(with good textbook). 

• It is the key to combine DL and face-to-face component in an ideal ratio. 
That ratio depends on a particular course in question. Also, when putting 
links on a page, instructor has to be aware of total amount of information. 

• Closeout of the modules is important. He is visual type, so he would like 
to occasionally see somebody talk about the issues in the course, even via 
video-stream or something like that. 

• On-line learning is good; it depends on subject matter. This course was 
really good. If there were more math, it would be too difficult. It depends 
on a course. It worked pretty well. 

• It’s important to capture well and to transform the knowledge into on-line 
content, but it’s important to be able to transfer that to both instructor and 
the students. On-line learning misses sharing of who knows what on 
important issues. Information has to be customized. SS3011 was better in 
on-line form probably because the students were forced to read all 
material. Technology had to be prepared better. 

• The instructor tried to get feedback from the students and was veiy opened 
to suggestions. The feedback is very useful for fixing the coxuse problems. 
Right now there is no perfect course on-line. There should be more 
mxiltimedia (applications, demos, interactive thing s), and some real-time 
chat, maybe even in voice transfer (some people are well spoken but can’t 
write very well). 

• Instructor should encourage students not to wait to the last moment (with 
studying). This course’s web pages were predominantly text-oriented. He 
had another course with much more graphical materials on web pages. He 
thinks that more interactive and graphical content should be put on 
SS3011 pages too. He is indifferent about real-time chat option, that 
would be like imitation of real classroom and DL wouldn’t gain much 
with it. 

• Instructor should meet with students in a classroom from time to time, at 
least couple of times, maybe in a round-table forum or in small groups. 
Such modification would facilitate learning process. Another problem is 
that students do not see such course as hi^ priority. List of all 
deliverables should be defined at the beginning of the course. That should 
be very usefiil. 

• Do not teach DL the same way as face-to-face learning. Information has to 
be well structured. 
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• Maybe forums should be real-time or both. Maybe groups should be 
changing, and they should meet in real time. Personal self-dedication is a 
necessity. 

• The course was OK. It woxild probably be more useful to a people outside 
of NPS - and it should involve more people. On-line section has to have 
meetings too, maybe 2-3, especially wrap-up discussion. 

• The only complaint she has (what also holds for some face-to-face 
courses) is that she likes to see at the beginning what will be covered in 
whole course. This was not visible in SS3011. That was the largest 
problem for her. She expects to be self-paced and would like to have at 
least a complete reading list. There is also a timing issue in forums and 
there was some ‘just in time’ studying. 

• He’d like more multimedia content, video stream and similar. It is very 
bad that many courses in NPS do not use any computer or Internet 
capabilities, 3D and similar! 


Ninth Question - Summary 

I reduced the student recommendations to their key messages to the instructor 
(ranked by frequency): 

• Fix all tech problems as soon as possible (three students). 

• Organize occasional meetings with students, every 2 weeks or so (three 
students). 

• Provide list of all deliverables/obligations at the beginning (two students). 

• Introduce some real-time or combined chat/forums (two students). 

• Give tougher quizzes. 

• Provide more feedback from the instructor (to individuals and to groups). 

• Develop a backup plan (in technical sense). 

• Provide better module closeouts. 

• Design more multimedia and graphical course content. 

• Find a way to force students not to study ‘just in time’. 

• Involve external subject matter experts. 

It was difficult to synthesize these categories of the student recommendations 
because some of the categories overlap. The recommendations address the following 
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issues: course web design - four responses; technical problems - four responses; 
communication between the students and the instructor — four responses; general 
organization of the course - three response. These are all very important elements in 
creating a positive learning environment for the course. 

10. Tenth Question 

The tenth question was: ‘This course was not only a space-systems course, but 
also a new experience for most of the students in this section. Looking at the course from 
this time distance and do you think you’ve gained some other benefits besides the 
course objectives themselves? Please comment.’ 

Here are the student comments to the tenth question: 

No 

• No, because most of students already had decent computer skills. Future 
IS going towards DL; EXPOSURE to it was main thing, kind of trial-and- 
error. 

• He already knows everything about DL. 

• Most of the students are good in IT^^. 

• This was his first on-line course, but he works a lot on-line. Without that 
knowledge, it would be harder. So, he didn't gain anything new. 

• He is very good in Web and PC technology, so he didn’t gain anything 
else. Maybe only new personal experience, and he may do something 
similar in the future (he works in IT area). 

• He had a plenty of on-line learning (10 courses) already. 

• Probably not. This is connected with issue of PC proficiency and with 
experience with the Internet. He is very good in that, so he didn't gain 
anything else significant. 

Only writing 

• For him personally, only challenge is to write better. He gained nothing 
new in technical sense; he is very proficient in web technologies. 


Infoimation Technology. 
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Yes 


• It was very different from the other on-line course (TW) she had. 

• She liked this way (of learning) and would like more. She has learned that 
there are more than one way for presenting information, and PLENTY of 
sources. The instructor did a great job. 

• He learned a lot about forums. He had computer proficiency before. 

• The course opened for him doors of new educational opportunities, and 
there are many (Net-G Navy program and other educational 
opportunities). He will certainly enter into such programs, and that's very 
good. This way (DL), educational/training possibilities can be much better 
focused, and possibilities and availability of educational opportunities are 
increased (and he needs them). In DL, it is also possible to choose only the 
courses one needs. 

• He learned that there is still a long way to go to a good learning in DL. 
People are different. We have to respect/recognize well subtle aspects in 
difference between DL and face-to-face learning. 

• He learned not to take a DL course if he is not ready for that. 

• Before coming to NPS, he hasn’t used the Internet much. Now he is much 
better in surfing, searching, solving problems. 

• It additionally opened her eyes, enhanced her awareness on available 
options. She was a little skeptical about learning effectiveness in such 
course, but now she remembers very well many things from die course (2 
months later). She is very interested in web technologies. It can be a very 
useful learning option, if it's done well. 

• DL is the future, now he works much easier within it, uses it without being 
tensed, but he still doesn’t have enou^ experience to compare it with 
other on-line courses. A course like this one can provide a ^dent with 
better understanding of future educational/professional opportunities. 


Tenth Question - Summary 

The responses to tenth question were: ‘no’ - seven responses; ‘only writing’ - one 

student response; ‘yes’ - nine responses. Analysis of these responses shows something 

very interesting. Practically all the students who responded ‘no’ used one key 

explanation: they are already very experienced in IT and in using the Web. Quite 

differently, the students who responded ‘yes’ had very different reasons for their answers: 
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new learning opportunities, very different course (from other on-line course(s)), enhanced 
awareness on options, forums, etc. Only one of these responses used the same 
explanation as the students who answered ‘no’. He explained he did not have much 
experience with the Internet. In other words, the first group of students (‘no’ responders) 
were more focused on only a single factor, while ‘yes’ responders had more diverse 
reasons. 

11. Eleventh Question 

The tenth question in the interviews was: ‘I asked you many questions in the four 
on-line questionnaires and in this interview. If you think there’s anything else I should 
know, but you haven’t comment it yet, please feel free to do it now.’ 

Only one student didn’t respond. Here are other comments, grouped into few 
typical sets of responses: 


Technical recommendations 

• It would be good if the server is set to e-mail replies to post automatically 
to original post writer’s address. The instructor was the key to the success 
of the course - she acted exactly in accordance with tendencies of 
decentralized leadership style. 

• A pilot-course had to be previously done in order to clean all‘bugs’. There 
was a lot of confusion with passwords. Too many questions in the 
beginning connected with technical problems, and if somebody is getting 
‘bad taste’ because of that, that’s bad for the course. Personally, he didn’t 
see that as major problem, he is used to things like that, especially in the 
military. 

• It’s certain that some beginner’s mistakes will not happen again. It all had 
to be tested better (pages, links), instead being fixed diiring the first two 
weeks. 

• Technical problems have to be solved before the be ginning of the course 
because they produce resistance in students. Fortunately, most of the 
students were in IT area and they are familiar with such problems. In 
general, it functioned well; it was designed intuitively. 


236 





Positive opinions 


She enjoyed the material. In on-line course it’s easier not to study 
regularly - slacking off. 

It was pretty decent course, and nothing important was lost with transfer to 
DL delivering. 

The course was OK. People were trying hard, and he personally gave 
many well-intentioned recommendations to the instructor. The students 
were pretty constrained in the learning process (module after module). 
One can’t go forward in advance, same as in ‘classical’ course. One can’t 
learn at one’s own pace. 


Other advice, comments, and a question 

Skeptic: On-line is more time consuming - that’s the key point. She is 
pretty skeptical about DL while on duty, even if it gives new 
opportunities. It all depends on student, how much time he/she invests into 
a course. 

More real-time events: She would prefer occasional real-time chat as an 
addition. 

Security: Security, log-ins and passwords have to be changed (they 
shoiildn’t be based on SSN). It’s privacy issue! 

‘Switch’: Many students said they haven’t learned much. He somewhat 
agrees, but that’s because they compared the course to resident courses. 
For DL one has to make a ‘switch’ in one’s own head. It is possible to just 
give minimum comfort in the course, but if a person has self-discipline 
and makes one’s own conclusions, potentials for learning are great. Links 
to the newest information is an advantage for DL; books do not have that. 
The textbook was very good too. 

‘Guinea pigs’: Face-to-face courses have an advantage while in the 
school. He felt that his class was kind of guinea pigs, and that produced 
resistance in many students. Everything was not technologically ready for 
the course. 

Quizzes and tests: He probably gained die most on the final exam: he had 
to collect it all, to organize, to formulate as a recommendation to a boss - 
that was most usefiil. He would recommend more good tests, not too easy 
- maybe tougher, could be e-mailed to students, and no multiple runs 
through the test. 

A question: He asked why the UCF model was selected for application in 
NPS. 



• ^Psychological factors*: On-line learning is very visual: some people 
don’t like to learn like that. Maybe learning style should be tested before 
sending a person to DL course. 

Eleventh Question - Summary 

The final comments reflected the different backgrounds and priorities of the 
students. Some addressed technical problems in the course; others were more focused on 
the learning-environmental or psychological factors of the learning process. Their 
impressions varied, but were more positive than negative. Most of the comments 
contained useful recommendations and suggestions for further development and 
improvements of the course. 

12. Interviews With The Students - Conclusions 

The interviews were individual, semi-structured, conducted four to six weeks after 
the course was completed, and only with students who volunteered. Out of thirty, 
seventeen students agreed to give interviews. None of the students with less than an A- 
final grade were among them. 

Eleven open-ended questions were asked in the interviews. They mostly 
addressed some important issues already explored through the on-line questionnaires. 

When asked whether they learned more firom the on-line than firom face-to-face 
course, eleven students disagreed; four students thought the learning would be 
approximately the same, and only two students agreed. Two explanations for disagreeing 
were dominant: the course content and design, and student motivation. One student 
mentioned environment as a key reason. The students, who thought of the learning 
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efficiency as the same, did mention differences between DL and face-to-face courses, but 
with equal opportunities and results. 

Asked about the usefulness of the forums, sixteen students saw them as ‘useful’, 
‘probably useful’ or ‘very useful, and only one student as ‘not useful. Even in opposite 
responses, the same argument was used - the argument that the forum posts represented 
the student opinions, not factual material. 

Ten students saw quizzes as useful, and six students as not useful. It seems that 
the difference foimd in opinions mostly comes from a different interpretation of the role 
of the quizzes in the course. Besides that, some students saw quizzes as useful, but 
anyway did not like them. 

Interactivity issues were addressed in four questions. Predominantly, students 

thought the amount of interaction with the instructor in the DL environment was less than 

in the face-to-face classes (six said - ‘much less’; eight - ‘less’). Analysis of their 

responses suggests that their key explanation was the absence of the instructor’s personal 

presence. Asked about the quality of interaction with the instructor, students most 

frequently thought it was good (seven students) or even better (two students), but a 

substantial number of students had the opposite opinion (five students - ‘less’). Lack of 

the instructor’s presence was again the main explanation in the latter case. Many students 

who thought quality of interaction was good or better mentioned some notable, but not 

overviielming problems in the DL course. When questioned about the amount of 

interaction between the students, eleven students responded that there was less such 

interactivity in the course compared to a face-to-face course. Those who though there was 

greater interaction, emphasized their explanations that students were forced or requested 
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to post to forums and thus to interact so much. Finally, students tended to see the quality 
of the interaction between students better than in a face-to-face class: eight said ‘better’, 
one - ‘little better’, and only three said ‘worse’. Several students though the quality was 
enhanced because they had to write postings, which stayed on a server until the end of the 
course. Some added that forum posts serve as references for further discussions and, 
therefore, the quality must be better. 

Most students would think carefully about whether to choose an on-line or face- 
to-face course: twelve students responded it depends on the environment, on the nature of 
a course, and/or on the instructor. Only five students answered ‘face-to-face’, but most of 
them added that it might depend on the environment or other factors. 

The students also gave interesting and applicable advice and recommendations. 
The recommendations were dealing with fixing the technical problems, organizing 
occasional meetings with the students, better defining what is expected firom the students 
(all deliverables), and other issues. 

In justifying their responses to the questions on the course as a new experience 
that includes some other benefits and not only the course objectives, students responding 
‘no’ were focused on fact that most students were proficient in IT technology, while 
others, who responded positively, thought about several factors. 

The last question asked the students to add other comment they wish. They 
responded with a wide spectrum of comments. The comments were mostly forthcoming 
recommendations for improvements of the course. Diversity of their responses reflected 
student backgrounds and priorities. 
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F. INTERVIEWS WITH THE INSTRUCTOR 

I met with the instructor several times to talk about different issues of the course. I 

planned four sessions exclusively for interviews about the course. The questions I 
intended to ask at a particular interview were sent to the instructor in advance. I divided 
the instructor’s responses by the main issues: 

The instructor’s preparations for the course 

As soon as the instructor learned she had to design and deliver the course in on¬ 
line environment, she started to organize material into directories, resources, sources of 
information, what took approximately several (maybe five) hours a week, for three 
months. From the end of January to March 2000 (8 weeks in total) she attended an DDL 
6543 (Interactive Distributed Learning For Technology-Mediated Course Delivery) 
course at the University of Central Florida. The course emphasized the development of 
web-based learning modules and on-line learning in general. During the course she 
developed a test-module (not included in the course later). From April to June 2000 she 
devoted 60% of her working time to course development. 

Commander Higgins did not have any previous web-development or on-line 
teaching experience. She has no previous teaching/instructing experience at all except for 
briefings, group workshops, and seminars. 

Preparations for an on-line course were a very creative process and it was a very 
positive and an intensive learning experience for her. 

On the‘UCF model’ 

She fully supports the ‘UCF model’. She believes that it is essential to enter into 

the transformation process and to ensure there is a supportive infirastructure for the 
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faculty. It is not ‘mechanical’ transformation into different media, but it is necessary to 
also start thinking in terms of how people learn and how best to use available tools to 
accomplish efficient learning. 

The UCF model is process-focused and includes continuous revising of the 
materials and obtaining feedback from students. I foimd significant similarities between 
these principles of the UCF approach, the explanations the instructor gave me, and the 
students responses about the course. UCF ensures that every DL instructor collaborates 
with am instructional designer who is a constant link between the instructor and several 
UCF faculty members. 


Initial problems in the course 

Scheduling problem: 45 students appeared on first day, because of the mix-up in 
the Registrar’s Office. She expected only 25 students. From this problem another one 
emerged - a problem with logins and passwords. 

Second, during the first week of the course the UCF servers, which hosted the 
course on-line materials, did not function because of relocation to new offices. NFS 
students were not in the UCF databases of students, which caused additional problems 
with logons and passwords. 

Firewall problem: some computers in NPS were configured differently than 
computers at the UCF campus, so unexpected problems with the firewall appeared. This 
was not discovered before the beginning of the course. 
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It took approximately two weeks to solve the problems described above. During 
one of the following weekends, the UCF servers were down, but during remaining time 
there were no similar problems. 

Some of these technical problems were generated exclusively because of the 
circumstance that the course server was located on the UCF campus. This course was one 
of the first non-resident, non-UCF courses hosted^® and supported by UCF, but taught at 
NFS. 


Key lessons learned for the instructor, in light of the initial problems are: 

• Complete all materials before the beginning of the course 

• Limit size of the group. For instructor-beginner, it should not be over 
fifteen to twenty students. 

• Assistant to the instructor, mostly for technical issues, but also for course 
design, content preparation, teaching issues, etc., is of essential 
importance. 


. Thinking about the beginning phase of the course, the instructor concluded she 
did not use the best approach: she set the stage and gave the roles to the students, and did 
not personally engage the on-line dynamics of the class as much. Because of problem 
solving process at the begiiming of the course, the first week was an excellent, valuable 
experience for her. Also, she needed additional advice about moderating the forum 
discussion, which she got firom UCF faculty and colleagues. In essence, the process of the 
instructor’s improvement continued. 


3® i.e,. UCF server hosted all on-line course materials. 
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Quizzes 


Quizzes were not planned to be included in the final grade, but as exercises that 
provided important feedback to the students. It was not important how many points a 
student received, but the students did not quite xmderstand the instructor’s intention. 
Students did not like the wotd ‘quiz’, so the insfructor decided to change it in the next 
iteration of the course to a ‘self-assessment’. 

The instructor was not aware that she would encounter so many technical 
problems with the quizzes. She was aware she would have an exceptionally high volume 
of work in the first iteration of the course. Therefore, she did not work on further 
development of the quizzes during the course, as much on any detailed further 
development of the quizzes. She decided to add more open-ended question to the quizzes. 

This was the first on-line course to most of the students 

The instructor was well aware that this course was the first on-line learning 
experience for almost all students. This fact coiild cause significant fiustration for them. 
She knew that this fact was not so important, but still was a significant issue for the total 
amount of concern, but still she had to deal with it. Due to this fact, she told them that if 
the number of problems were too great they could drop the on-line section and enroll in 
the face-to-face section of the same course. For various reasons, ten students decided to 
do so: some because of their frustrations and some without any specific reasons. After 
that transfer, thirty-three students remained in this class. 

After the first week of the course, comments and recommendations of the 
remaining students to the instructor were very constructive and useful. The instructor 
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concluded that after the first week the students demonstrated effective learning and 
proved to be able to use the course software. 

Grading criteria 

The instructor had a grading sheet for each student. Final grades were based on 
the following: 

• ‘Doing everything’ in sense of reading module materials and participating 
in every forum on a regular bases Forums - initiative and responding 

• Answering the quizzes regularly (number of points in quizzes was not so 
important) 

• Final paper 

According to the instructor’s estimation, the first two criteria consisted of 
approximately 50% of the final grade, and the final pqier the remaining 50%. 

For grading the final papers, she had three criteria: 

• Physical influences on space systems 

• Impacts of commercial space operations to DoD 

• Impacts of policies on accessing space systems 

. Good elaboration in all three areas in the final paper would mean an A grade for 
the final paper which was 50% of the final grade. 

Since this thesis research questions address learning effectiveness, and final 
grades were only objective indicator of learning outcomes, the instructor’s explanation of 
grading criteria is very important for interpretation of the results. 
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The student feedback to the instructor 


The instructor thinks that students had different overall affective impressions 
about the course partially because they were ‘different types of people in personal 
characteristics relevant to one’s adaptation to a DL environment’. 

Approximately half the students visited the instructor in her office. Most of them 
had a very positive opinion of the course. Very often they stated that while they were 
resident IsfPS students, they would prefer a face-to-face compared to the on-line course. 
However, they liked experience of an on-line course. Their responses in SOF were not 
surprise for the instructor, because she already received large amount of feedback from 
students. 

The initial quality of student inputs in the forums far exceeded the instructor’s 
expectations. The forum posts were very challenging to the instructor because she had to 
find optimal timing and way to moderate, facilitate, lead, and control the threaded 
discussions. On the other hand, trying to have complete control over that process would 
not be good. 

Several students suggested real-time on-line chats or similar kind of interaction. 
Near the end of the course, their suggestions emphasized more the need for additional 
classroom sessions, maybe once a week or every 14 days. The instructor felt that need 
also. Such change would make this course a hybrid, not entirely web-based course. That 
could not be acceptable for all future non-resident students. 

The instructor saw some of the students in the group only once during the first 
session before the on-line course actually began. However, even before the course was 
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over, the next quarter’s students started to ask her about the on-line section and expressed 
significant interest. Another valuable source of feedback were SOFs. 

What the instructor decided to improve for second iteration of the course 

Other researchers in this area usually think that an instructor ‘stabilizes’ in the 
third iteration of the on-line course they teach. Based on experiences fi:om the first 
iteration of the course, this instructor intends to: 

• Increase amount of interaction with the students 

• Reduce technical problems as much as possible 

• Avoid last minute content preparations 

• Rename quizzes to ‘self-assessment’ and improve them 

Other lessons learned 

The course was more time consuming for her than she had expected. Probably 
more imexpected problems appear in this type (on-line) of course than in face-to-face 
comses. The first iteration of an on-line course should not have many students, and the 
schedule should not be too tight Especially at the beginning of the course, the instructor 
needs to have more reserve time, especially in terms of unexpected technical problems. 

The instructor said her emphasis was on course implementation, not on course 
design, which includes redesigning after every good suggestion or recommendation firom 
the students. This would not be rational. 

Student perception of the course load in an on-line course follows different logic 
than in face-to-face courses, and an instructor has to be careful about this, especially 
when teaching such a course for the first time. Also, e-mail and forum communication 
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demands additional caution, which is different from the usual personal, verbal and 
nonverbal communication between an instructor and students. 

Some students were anxious about the on-line course in the beginning. She had a 
line of the students in front of her office. 

She plans to prepare more multimedia materials for the next iteration of the 
course, but not many because of the bandwidth problem. Some students do not have high¬ 
speed coimections. 

The instructor also added that not having previous face-to-face teaching 
experience might be an advantage for her. She did not have to unlearn face-to-face 
versions of the course and face-to-face methods of instruction did not interfere with the 
on-line teaching methods. 

At the end, the instructor added that the policy on DL at NFS is changing in terms 
of enabling more people to attend DL courses in the Navy, not only in the fleet on 
carriers and ships, but also on shore. Also, the Navy scheduled significant improvements 
in bandwidth on ships. In other words, DL gained increased organizational support at 
NFS and in the Navy, which was an additional acknowledgement and motivator to the 
instructor of this course. 
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V. DISCUSSION 


This thesis was initiated by die need for evaluation of one of the first on-line 
courses taught at NFS. That need was not only a consisting part of ‘UCF model’ of DL, 
but also an important step of permanently evolving instructional design process, and a 
part of academic and organizational culture in NFS - what is primarily embodied in 
mandatory Student Opinion Forms. 

Studying a single case of relatively small number (thirty) of SS3011 on-line 
course students, I collected data fi'om nine main different sources: four on-line 
questionnaires, SOFs, student on-line tracking, final grades, interviews with the students, 
and interviews with the instructor. Even though it was not possible to provide 
sophisticated level of data analysis (because of small sample, mostly ordered categorical 
data, and because large part of instruments was administered anonymously), many 
relations can be discussed and conclusions possible to drive. 

Before integrating all findings and their implications and answering this thesis 
research questions, I will discuss results of each one of the data sources I used in this 
research. 

The First on-line questionnaire (A-1 or ‘The Basics’), administered at the 

beginnmg of the course, provided a quick overview about the students (quarter, service, 

pay grade), about their relevant previous experiences (on-line courses, space-related 

courses), computer and Internet user proficiency, expectations, and concerns. It may be 

said that as first of four on-line questionnaires it had secondary role of preparing the 

students for ‘more serious’ questionnaires to come. However, insight into responses and 
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into time the students needed in average to fill up a questionnaire on-line confirmed that 
they had no problems in completing them. 

Finding that the students attending later quarters tend to visit the course homepage 
fewer times, to send fewer posts to forums (what includes follow up posts too) can be 
interpreted with important fact every NFS student has to learn - with the need for 
selection of information and for time management. Very likely, more ‘senior’ students 
tend to be more selective and more careful with their time, so this tendency seems 
reasonable. 

The data indicated that vast majority of the students had neither previous 
experience in on-line learning nor experience in space-related courses. That is to say, this 
case study is about first application of an on-line course in Space Systems Academic 
Group, NFS, first time by this instructor, and as first DL course for almost all of these 
students. Briefly, this was a pilot project of NFS course transformed to DL using ‘UCF 
model’. 

Most of the students perceived themselves as pretty proficient in using computers 
and in on-line search techniques (what was expected, having on mind their curriculum), 
and most of them expected to improve their learning through the on-line course. 
However, those with lower proficiency level and with lower on-line search skills had 
higher expectations for improvements, while more computer proficient students tended to 
read larger number of forum articles. According to these relations, it is doubtable that the 
question on expected learning improvement closely corresponds to ‘self-efficacy’, 
probably because of formulation of the question (it does not address any visible criterion, 
but only unclear term ‘improve learning’). 
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Self-perceived on-line search technique also has interesting correlations, &st - 
with self-perceived computer proficiency, and second - with number of different course 
pages visited. The students more skilled in searching on-line visited fewer course pages 
probably because they find easier what they were looking for; they select information and 
save time. 

Open-ended questions to this and other questionnaires provided very interesting 
qualitative data, which I analyzed using grouping of responses by meaning, i.e., by 
message(s) the responses were stating. In First questionnaire, student comments (last 
item) were more positive than negative, with few statements of personal preference of 
traditional classroom and few complaints about large forums. 

In general, First questionnaire provided pretty positive picture and insight into 
relation between expectations and self-perceived proficiency before the course. 

As Suchan and Crawford (1998) recommended, this research included relevant 
measures at different points in time (i.e., at different time points within the course), firom 
orientation phase (First questionnaire and partly A-2 questionnaire) to the period after the 
course was completed (interviews witii the students). This way, important aspects would 
not be missed. Also, I collected many qualitative data using on-line questionnaires and 
interviews. The instructor of SS3011 on-line course followed many other 
recommendations the authors listed in their paper. 

The second questionnaire (A-2) was administered during second and third week 
of the course, after the students got clearer impression about the course. It was 
anonymous and had 26 items: 17 were five-point scales and the remaining were open- 
ended questions. 
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This questionnaire addressed much wider spectrum of issues than the first one. 
Responses on questions about comfort level mostly provided very desirable distributions, 
but on several issues there were individual students that had some problems: four 
students were ‘somewhat imcomfortable’ in setting their systems; one student ‘extremely 
uncomfortable’ in accessing the Internet; two students uncomfortable (one ‘extremely’ 
and one ‘somewhat’) in word processing; four students imcomfortable in posting to 
forums, etc. However, knowing for significant organizational and technical problems at 
the beginning of Ihe course (reported by the instructor in the interviews), such 
distributions of responses were smprisingly positive. 

Largely neutral responses to questions about comfort with technical support and 
help indicated that many students didn’t need to use that help or were not aware of that 
option. 

Their responses to open-ended question about needed techniced help were mostly 
about posting to the forums, what was currently central technical problem. 

Interpretation of the student responses to eight statements about comfort level 
with use^* of different features in the course (forums, e-mail, quiz, links, course 
overview, etc.) is not quite clear, because the question is whether these items represent 
student comfort level or effectiveness of respective features wi thin the comse. It depends, 
I would add, on how we define effectiveness. Of course, the student responses caimot be 
objective indicators, but their subjective views of supposed effectiveness. 


These questions were targeting the way how these features supported learning process in the course, 
not how the students felt comfortable using diem. 
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Analysis of correlations between the variables in this instrument showed that 
questions asking abut comfort levels with activities and on-line activities present in 
the course (first five questions in the instrument), and the question about comfort levels 
with features used in the course, do correlate with at least six, but mostly with ten or 
more other variables. Several correlations were over 0.70. Four questions asking about 
comfort level with different forms of technical support had much fewer and in average 
much lower correlations - ranging fi'om one to four statistically significant correlations 
with other variables. The interpretation is much harder in this case. Many responses were 
neutral for simple reason that the students have not used or were not aware of possibility 
of such support. 

Very interesting were responses to the question asking them what they liked about 
the on-line course. I grouped nineteen responses as ‘time managements flexibility and 
accessibility’, what is very similar to findings of Moskal and Dziuban (2000). 

I grouped students responses to the question about their concerns to: technical 
problems; uncertainty about comparability of DL to face-to-face courses; learning pace 
and organization; grading; and miscellaneous. This list of their concerns exceeds main 
problems the instructor outlined in interviews, viiat points to the benefits of this sort of 
feedback collection firom DL students. Their responses reflected not only current 
problems they were facing in the course, but also issues every student meets - issues of 
self-organization and self-discipline (‘learning pace’) put into DL context. 

Student responses to the question asking them how would they build and teach an 
on-line course were mostly recommendations for improvements of instructional methods 

32 infonnation technology. 
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or for fixing technical problems. However, six students would not change anything from 
this course. 

Module 14 questionnaire was administered close to the end of the course, asking 
the students three sorts of questions: how well the module 14 learning objectives were 
met; comfort level with the course features and relevant technical issues; and one open- 
ended question, asking for additional thoughts. 

First seven items gave positive picture on module effectiveness in terms of met 
learning objectives. Only exception was the question on interaction with a subject matter 
expert (what made this module exceptional), because the number of neutral responses 
was very large compared to other six questions in that part. Possible explanation is 
students who chose neutral answers probably missed the opportunity to interact with the 
expert. 

Responses to these seven statements significantly correlated with other variables 
from the instrument, but some also with some of the on-line student tracking indicators 
and few variables in First questionnaire. 

The second group of questions was parallel to identical questions in Second 

questionnaire, so comparison of same variables in different time points in the course is 

made. Distributions were very similar, showing some improvements over the course. 

Most of them had several significant correlations, mostly with other question from this 

group and with on-line tracking indicators. Very interestingly, two questions (about 

comfort level with quiz feature used in the course and about comfort level with word 

processing) correlated with final grades too. All on-line tracking indicators had 

significant had significant correlations with other variables, some with several variables 
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from different instruments, wbat xmdoubtedly confirmed they have to be considered as 
important data source in DL effectiveness studies. 

Comments added at the end of this questionnaire were mostly positive and 
constructive, but there were some hybrid comments, some complex, and some skeptical. 

Final survey, the last of four on-line questionnaires in this research, were 
administered after the completion of the course, before the students got their final grades 
- so possible impact of grades to student responses was avoided. 

First six questions followed the course learning objectives and most of the 
students responded positively. Only the question asking about usefulness of final exam in 
terms of helping a student to synthesize the concepts learned in the course had four 
negative responses. However, what is especially intriguing with these questions are 
correlations: most of significant correlations seemed to be in average most numerous and 
highest with other of these six questions, but that was not absolute rule. Correlations with 
some other questions were also significant, e.g. with questions on flexibility, questions on 
quality of interaction with the instructor, questions on usefulness of some of the features 
used in the course, or question on liking some of the course on-line features and 
activities. These correlations may seem irrelevant if one reviews theories mentioned in 
first two chapters, but they may raise new question and new, unexpected constructs in 
description and analysis of DL process. They are statistically significant and they seem to 
discover new relations, for which we still don’t have tiieoretical terminology. 

Seven next questions asked the students about usefulness of the following course 

features or elements: guidelines (two questions); textbook; on-line module readings; the 

links to websites; forum exercises; and quizzes. The distributions of these variables were 
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strongly positive, but some of them had notable number of negative responses too, those 
addressing usefulness of forums and quizzes. This was another confirmation that the 
students significantly differ in their interpretation and perception of forums and quizzes 
more than in interpretation and perception of other course features. Most of these seven 
variables had very few significant correlations: fi'om one to four. Only exception was the 
question about usefulness of module readings, with ten significant correlations. Again, 
some of correlations were not expected and are hard to explain. 

Liking of a course feature correlates with an opinion on its usefulness, but it is not 
the same. For instance, correlation between the responses about liking the forums 
assignments and the responses about usefulness of forum exercises was p = 0.62; 
correlation between responses about liking the textbook readings and responses about 
usefulness of the textbook was p = 0.64; the same for module readings was p = 0.54; and 
for website links p = 0.56. These are statistically significant but mild correlations. In 
cases of forum exercises/assignments and the textbook, these correlations were highest 
among very few other significant correlations, but for module readings and website links 
they were not. Moreover, in latter cases some other, unexpected correlations appeared to 
be higher, and there were much more significant correlations. The question asking about 
liking the module readings had ten, and the question asking about liking website links 
had nine significant correlations with other items in the instrument. Interpretation of this 
difference in numbers of correlations would be that latter two correlations (module 
readings and links) were more interrelated to many other factors of the course learning 
process, while examples like textbook of forum exercise represent more specific, 
relatively isolated sort of activities within the course context (the textbook is not on-line 
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feature, but is very important element of the learning process, while website links is the 
course material design and content feature). Among others, responses about some of 
aspects of interactivity (amount of interaction with other students and quality of 
interaction with the instructor) were also significantly correlated with two described 
features (module readings and website links). 

The student responses to open-ended questions were very constructive and 
provided interesting recommendations. They had positive opinions on instructional 
strategies used in the course, but most of them didn’t think the course fimdamentally 
changed their approach to learning. 

From responses to A-2 questionnaire we found that the students saw the flexibility 
(in sense of time management) to be very be beneficial. In Final survey the research went 
one step further and added two questions to determine vdiether that flexibility was 
beneficial only within the course or in terms of the student success in other courses. The 
student responses confirmed that the latter was more significant - the time flexibility was 
beneficial to this sample of students mostly in order to succeed in other courses. 

The interactivity issues were among the most interesting points in this research. 
The student responses in Final survey showed that most of the students drought that 
amount of interactivity between the students and the instructor was less (compared to a 
face-to-face course). There was a similar trend concerning quality of interaction between 
the students and the instructor, but number of neutral responses clearly dominated. 
Similarly, the student responses on amount and quality of interaction between the 
students indicated decrease, with significant number of neutral responses and with few 

opposite opinions. This does not match findings of some other researchers. Wang, 
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Dziuban, and Moskal (2000) found that UCF students saw quality of interaction with the 
instructor increased in on-line courses, but findings about quantity were not so clear. 
Interactions with other students were perceived as slightly increased in quality, but it was 
not clear about the quantity (what also partially depends on way how we ask the students 
about their opinion - on-line or in mailed siurey). According to Hartman, Dziuban, and 
Moskal (1999) and Moskal and Dziuban (2000), vast majority of faculty saw the amount 
and the quality of interaction significantly increased in comparison to face-to-face 
courses, especially in cases of fully on-line courses. However, the authors (Moskal and 
Dziuban, p. 26) anticipated that relation knowing that the faculty was requiring students 
to post to the forums, i.e., they were forcing students to interact. Similar was in the case 
analyzed in this thesis, and some of the student comments gave the same explanation. 

The student responses to Long-Dziuban Reactive Behavior Protocol were 
surprising if compared to findings of other authors (Dziuban, Moskal, and Dziuban 
[1999]). In case of sample in this thesis research, much more students (50%) belonged to 
Passive Independent type (compared to 12% of UCF students in mentioned study). 
However, supposed differences between two populations, selectivity, environment, and 
other possible factors do not allow conclusions without additional and more focused 
research. Same holds for three found significant correlations. 

Student Opinion Forms, administered on official NPS forms at the end of the 
course, also provided interesting information. Statistical analysis of responses was limited 
because of ‘No comment’ responses, which had to be excluded firom calculations. 
Besides that, two questions seem not to be appropriate for DL environment and NPS 
should make changes in SOF forms to eliminate this problem. 
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Finding from First questionnaire about little importance of whether a student 
attend first or sixth quarter was confirmed in SOFs - this item did not have any 
significant correlations with other variables in this instrument Same holds for number of 
credit hours. 

All distributions were dominantly positive about different aspects of the course 
(respectively), with sporadic negative or neutral responses. Large number of correlations 
between the responses to SOF questions/statements was statistically significant, but they 
were reflection of targeted constructs the instrument was intended to assess. I would add 
that this thesis and other relevant follow-up theses and research might provide 
suggestions and recommendations what could be added, removed or changed in SOFs for 
DL courses to serve their purpose even better. 

Some of the correlations provided additional in-dqpth insights into relations 
within DL learning context, like positive correlation between the student views of the 
instructor’s course objectives made clear and the students feeling stimulated in the 
subject area. Such findings are closer to general pedagogical issues than to DL specifics. 

Student on-line tracking data were one of surprises in this research. All tracking 
indicators had significant correlations with odier variables: with other tracking indicators; 
with variables from First questionnaire. Module 14 questionnaire; and with final grades. 

For most of the indicators, correlations with comfort level with quiz feature, 
posting to forum, accessing the Internet, and with some other variables, were 
exceptionally high. The instructor especially appreciated the student originality and 
initiatives in forums postings, what was proved by correlation between the number of 
original posts and the final grades. 
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The distributions of total number of hits or number of forum postings had wide 
ranges between individual minimums and maximums. There were over 500 postings in 
forums, and their length varied from just a few lines to several paragraphs of text, so it 
was not easy to synthesize their content. Also, brief analysis of dates and time of posting 
the forums messages by time in day and by day in week confirmed convenience and 
flexibility of DL course. Unfortunately, in this research it was not possible to examine 
relations of on-line tracking indicators with some other variables, like Long-Dziuban’s 
reactive behavior types, what is one of possible directions for future research. 

Conclusion from these findings about on-line tracking is clear: on-line tracking 
indicators are relevant in DL evaluation and can significantly contribute not only in 
evaluation efforts but also in further research of DL effectiveness factors. 

The instructor stated in one the interviews that the student on-line activities were 
one of criteria for final grade. This is confirmed with significant correlations between 
final grades and all on-line tracking indicators. Final grades also had two other significant 
correlations, with comfort levels in quizzes and with comfort level in word processing. 
However, methodological limitations because of small sample and non-normal 
distributions indicate need for more thorough research, with large sample. 

Interviews with the students were conducted for several reasons: first, they 
provided more qualitative data; second, it was possible to conduct them after the course, 
when the students were free of final exams pressure, and when they had substantial time 
distance from completed SS3011 course; third, my choice of the interview questions was 
inspired by the most controversial items from all four on-line questionnaires, by the 
research questions, and by the instructor’s recommendations. 
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The most of the student did not think they learned more from on-line course than 
they would learn from a face-to-face course. However, they saw forums as useful and 
quizzes as relatively useful, and it seems that die differences in opinions occurred mostly 
because of different interpretation of the role of these features in the course. 

Asked to compare the interactivity in the DL course with interactivity in face-to- 
face courses, they thought the amount of interaction widi the instructor decreased, but the 
quality relatively increased or was the same. Similarly, most of them thought that amovmt 
of interaction between the students decreased m the on-line course, but the quality 
increased. 

In a situation to choose, most of the students would consider taking on-line 
instead of a face-to-face course, but their decision would depend on the environment, the 
nature of a course, or on the instructor. 

At the end of the interviews, the students provided many interesting comments 
and recommendations, which predominantly reflected their backgrounds and priorities. 

The instructor provided many important statements about; her preparations for the 
course; her view of the ‘UCF model of DL’; initial problems in the course; the grading 
criteria; the student feedback; needs for improvements in the course; and lessons learned. 
Her comments and explanations put new light on several issues present in the student 
responses in the questionnaires and die interviews, and without this part of the research 
many wrong conclusions could be made. 

Having on mind all organizational and technical problems at the begin ning of the 
course she described (exceptionally large number of the students appearing on first class. 
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firewall problems, software ‘bugs’, problems with UCF server being down, etc.), many 
negative student responses do not seem to address the curse itself, but the described 
circumstantial problems. However, there were also many positive reactions from the 
students expressed in different occasions but not explicitly visible in the data collected in 
this research. 

Teaching the on-line course for this instructor was creative, demanding, and 
continuous learning process, and she provided several lessons learned during the course, 
which are directed to some organizational issues, the course design and content issues, 
feedback to the students, etc. 

The most astonishing fact from the interview with the instructor was that she did 
not have any previous experience in on-line teaching, but most of the data from this 
research confirmed that the course in general succeeded, in spite of this fact and all other 
problems I mentioned. Many experienced DL instructors assess that most often for a DL 
instructor is necessary to go through three iterations of teaching a DL course to become 
much more comfortable, stabile, and successful. Organizational support is also important 
for an instructor’s success, what also takes time and efforts in organizations where DL 
education is still not part of culture. 

Looking back to the theoretical models presented in introductory chapter and in 
literature review, I must emphasize that the case analysis results cannot directly provide 
confirmation or results denying such models. Case study made on small sample and in 
specific circiunstances caimot ensvure firm data that could be generalized to such a model. 
However, quantitative and qualitative analysis confirmed that several groups of factors 
could be identified on the bases of their relations and qualitative analyses. 
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Learner’s characteristics from Baldwin and Ford’s model of learning design are 
certainly present and very important group of factors, what has support from various data 
in this research. One of factors in this group that should be considered in future research 
is Log’s theory of reactive behavior patterns, but there are some other theories (like 
theory of learning styles) and constructs (efficacy, organizational commitment) that may 
play important role in DL effectiveness. Students varied by motivations and expectations. 
The research confirmed that more computer proficient students could have lower 
expectations to gain from a DL course. However, the responses depend on what has 
really been asked. In this research, the question about student expectations asked about 
improvement in learning, not general expectation. 

Learning design was also addressed in large number of research items. The 
forums, website links, module objectives, guidelines, and numerous other design 
elements were examined. The data confirmed that there are many complex 
interrelationships between these and other issues during the course, including some 
unexpected relations, not known from existing literature. Qualitative analysis added more 
issues to be considered in future DL courses. 

Work environment factors in this case were more visible from the instructor’s 
fiian from the student side. There are many new initiatives in U.S. Navy supporting DL 
training and education, but that issue was not directly addressed in the research. The 
majority of students didn’t know would their future duties include area covered by this 
course, so examining the role of opportunity to use the learned from the course didn’t 
have much sense. 
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One of the findings in this research is probably the key role of interactivity, which 
in fact means feedback loops firom the instructor to the students and vice versa, but also 
between the students. Role of the interactivity/feedback is mentioned in many other 
articles and addressed in several researches, but is not visible in these models. 

Kraiger’s classification of learning outcomes had indirect support in the data: the 
research found that liking a course feature or an element of the course content (affective 
outcome) and being aware (i.e., know, be convinced) of its usefulness (cognitive 
outcome) is not the same, but correlates. Improvements in using on-line activities 
(posting to forums, using links,etc.) represented skill-based outcomes, but not in terms of 
the course objectives, only in terms of skills necessary to successfully learn in DL 
environment. 

The analysis of the student responses to questions about meeting the course 
objectives and of final grades (which was only objective measure of cognitive outcomes 
and in fact of coiurse effectiveness) showed very complex relationships. Two sorts of data 
appeared to be most correlated with the grades and with the student responses about 
course objectives: on-line tracking indicators, and some ‘general technical skills’ 
indicators (comfort level in using forums, in sending e-mail, etc.).Only conclusions I can 
make 

I would conclude that Belanger’s firamework for DL was closest to the reality of 

learning process within this on-line course. This model assumes bi-directional relations 

between the ‘players’ in the fiumework, I would suggest that crucial role of 

interactivity/feedback be explicitly stated in the model. Society impact and learning 

outcomes desired by the society are implicitly present in the case I examined, but there is 
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no need to comment it. All three sides - the learners, the instructor, and die institution 
(here the large part of institution’s role was covered by UCF) - were present and 
interacted during the course. 
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VI. CONCLUSIONS 


In spite of the fact fliat this course was the first taught by the instructor and in 
spite of organizational and technical problems at the beginning of the coinse, the web- 
based SS3011 course design met desired learning outcomes. 


The student expectations and motivation were predominantly positive, but not in 
consensus: small students were skeptical, and would rather attend a face-to-face course. 
On-line reactive behavior patterns found in this sample significantly differed firom other 
studies, but there is no enough support in data for firm interpretation. The student 
affective reactions were predominantly positive. Utility reactions correlated with 
affective, but not strongly. Only slight positive changes over the course were detected. 


Main lessons learned firom this course are: 

• Technical problems should be fixed as soon as possible, best before the 
course. 

• The forums should be limited by number of participants and by size if 
posts. 

• Feedback firom an instructor is very important, but the other dimensions of 
interactivity (students to instructor; between the students) are also crucial 
for the success of an on-line comse. 

• An on-line course can provide desirable flexibility and convenience of use 
to resident students, but it would be more effective in non-resident 
environment. 

• A coiurse evaluation is an integral, important part of DL education. It 
shoidd be conducted in different time points of the comrse, using 
qualitative, not only quantitative methods too. 

• For an instructor, especially if it’s his/her first DL teaching experience, 
DL course is very time-consuming, but creative process with lots of 
learning. 
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Other important findings of this research: 


The course organization was endangered by major scheduling problem at 
the beginning, but later it went on successfully. 

The pedagogy used in the course was very effective for most students, but 
small number of students had problems. Long’s typology or some other 
approaches to individual differences relevant for DL learning could be 
used as helping tool in providing such students an adequate instructional 
counseling or other needed support. 

The technology used in this course had problems at the beginning, what 
underlines the need for adequate technical and organizational support to a 
DL instructor. Technology should be ‘invisible’, so the learning process 
would be more efficient and attractive to students and faculty. 

This research opened many possible directions for future research. Many 
correlations between the variables, trends, and other indications found in 
this research cannot be interpreted; some of the existing constructs should 
be reexamined, and may be some new created. The most important is to 
understand the complexity and interdependence of many factors relevant 
for effectiveness of a DL comse. 
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APPENDIX A. A-1 QUESTIONNAIRE 

NUMBER OF QUESTIONS: 25 


THE BASICS 

Your Demographics and Studying Strategies 

The on-line Distributed Learning (DL) environment of SS3011, Space Systems 
Technology and Applications, is new to DoD and Naval Postgraduate School. We would 
like to gather data through the course that will assist us in improving future DL courses. 

In an on-line course-learning environment, teaching strategies, in particular, are 
implemented in a manner that will enable students to take an active role in their learning. 
To achieve this goal, courseware developers have begun to think about ways to foster 
Teamer-centered’ on-line instruction. This process has included not only looking at 
demographics of the adult learner, but also die professional/career goals and objectives as 
well as the studying strategies of those taking electronic courses. These types of data can 
assist courseware developers and researchers to ensure that , ‘teaching practices’ are 
responsive to the needs of the on-line learner. Your help in responding to the following 
questions will aid us in this endeavor. 

Question 1 

Your e-mail address: 



Question 2 

What is your student status for this course? 
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r 1. Full-time Student at NFS. 

c 2. Full-time Military, Auditing Course, 

r 3. Other. 

Question 3 

If you answered ‘other’ to the prior question, please enter your status here: 



Question 4 

About your academic and professional careers... 

What is your Service? 

r 1. USN 
r 2. USMC 
r 3. USA 
r 4. USAF 

r 5. OTHER - complete the following question. 


Question 5 

In the answer to the previous questions was ‘other’, please fill in your service here. 







Question 6 

What is your pay grade? 


r 1. 01 

r 2. 02 

r 3. 03 

r 4. 04 

r 5. 05 

r 6. Other 


Question 7 

What is your Designator/MOS? 


Question 8 

Including this quarter, how many academic quarters have you been enrolled at NPS? 

Answer: 



Question 9 

What is your curriculum? 

Answer: 
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Question 11 

Briefly list your previous duty stations and billets. 



Question 12 

What is your next duty of assignment (if known)? 

Answer: 


Question 13 
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List prior space system-related tours (e.g., commands/jobs). 



Question 14 

List previous university-level space systems-related courses. 



Question 15 

About your experience, to date, in taking an on-line course... 
Title of on-line course you are enrolled in for this survey; 


r 1. SS 3011 Space Systems - Technology and Applications. 

Question 16 

Have you experienced an on-line course (or coinrses) that used a web site to do/complete 
coursework? 

r 1. No 
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r 2. Yes 


Question 17 

If you answered ‘Yes’ to the prior question, enter the niimber on on-line courses you have 
taken. 

Answer: 

Question 18 

How often do you expect to use this course web site? 

r 1. 3-5 times per week, 
r 2. 2 times a week. 
r 3. At least once a week, 
r 4. I don't know. 

Question 19 

The computer you plan to use most often to access the course is located: 

r 1. InmyBOQroom. 
r 2. In my house, 
r 3. In a computer lab on campus, 
r 4. At my place of work, 
r 5. In the library. 

r 6. In a shared area of my living quarters (e.g., home, BOQ, apartment, etc.), 
r 7. Other - complete following question. 
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Question 20 

Please type in location/s, if the computer you plan to use most often is some place not 
listed in &e prior question. 



Question 21 


On a scale of 1 to 5 where l=novice and 5=expert, how would you rate your current 
proficiency in using computers? 

r 

1. 1 

r 

2. 2 

r 

3. 3 

r 

4. 4 

r 

5. 5 

Question 22 


On a scale of 1 to 5 where l=novice and 5=e3q)ert, how would you rate your on-line 
search resources and techniques? [Five-choice scale, firom 1 to 5,] 


Question 23 

Please indicate the extent to which you agree with the following statement; 
I expect this on-line course will be valuable and improve my learning: 
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r 1. Strongly disagree. 

c 2. Somewhat disagree, 

r 3. Neutral, 

r 4. Somewhat agree. 

c 5. Strongly agree. 


Question 24 

Please indicate the extent to which you agree with the following statement: 

I am concerned about my ability to use the web-based materials in this class: [Five-choice 
scale, from ‘Strongly disagree’ to ‘Strongly agree’.] 


Question 25 

Any additional comments, concerns, or suggestions are welcome: 









APPENDIX B. A-2 QUESTIONNAIRE 


NUMBER OF QUESTIONS: 26 


You Have Completed The Orientation Portion Of This Course. 

How Are Things Going So Far? 

Question 1 

How would you rate your comfort level at this point with setting-up your system? 

r 

1. Extremely uncomfortable. 

r 

2. Somewhat uncomfortable. 

r 

3. Neutral. 

r 

4. Somewhat comfortable. 

r 

5. Extremely comfortable. 

Question 2 



How would you rate your comfort level at this point Avith accessing the Internet? [Five- 
choice scale, from ‘Extremely uncomfortable’ to ‘Extremely comfortable’.] 


Question 3 

How would you rate your comfort level at this point with word processing? [Five-choice 
scale, from ‘Extremely uncomfortable’ to ‘Extremely comfortable’.] 


Question 4 

How would you rate your comfort level at this point with posting to the forum? [Five- 
choice scale, from ‘Extremely imcomfortable’ to ‘Extremely comfortable’.] 


Question 5 
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How would you rate your comfort level at this point with sending electronic mail? [Five- 
choice scale, from ‘Extremely xmcomfortable’ to ‘Extremely comfortable’.] 


Question 6 

In what areas of the on-line technologies (system set-up, accessing the internet, word¬ 
processing, posting to forums) do you still need help? 



Question 7 

Please provide any additional thoughts. 



Question 8 

How would you rate your comfort level at this poiut with the technical support provided 
for using the WebCT software? 

r 1. Extremely uncomfortable, 

r 2. Somevdiat imcomfortable. 

r 3. Neutral, 

r 4. Somewhat comfortable, 

r 5. Extremely comfortable. 

Question 9 
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How would you rate your comfort level at this point with the technical support available 
to you through yotir Internet service provider (ISP)? [Five-choice scale, from ‘Extremely 
uncomfortable’ to ‘Extremely comfortable’.] 


Question 10 

How would you rate your comfort level at this point with the technical support available 
to you through the university ADP office? [Five-choice scale, from ‘Extremely 
uncomfortable’ to ‘Extremely comfortable’.] 


Question 11 

How would you rate your comfort level at this point with the technical support available 
to you on CD-ROM? [Five-choice scale, from ‘Extremely uncomfortable’ to ‘Extremely 
comfortable’.] 


Question 12 

What technical issues (e.g., those related to WebCT, Internet Services, Local ADP, CD- 
ROM, etc.) do you still need to resolve? 



Question 13 

Please provide any additional thoughts. 
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Question 14 


How would you rate your comfort level at this point with the forum feature being used 
to teach this course? [Five-choice scale, from ‘Extremely uncomfortable’ to ‘Extremely 
comfortable’.] 


Question 15 

How would you rate your comfort level at this point with the email feature being used to 
teach this course? [Five-choice scale, from ‘Extremely uncomfortable’ to ‘Extremely 
comfortable’.] 


Question 16 

How would you rate your comfort level at this point with quiz feature being used to 
teach this course? [Five-choice scale, from ‘Extremely uncomfortable’ to ‘Extremely 
comfortable’.] 


Question 17 

How would you rate your comfort level at this point with the links (i.e., additional web 
sites) feature being used to teach this course? [Five-choice scale, from ‘Extremely 
uncomfortable’ to ‘Extremely comfortable’.] 






Question 18 


How effective was the course overview? That is, to viiat extent did the course overview 
help you to understand what you would learn in the ‘Space Systems - Technologies and 
Applications’ on-line course. [Five-choice scale, from ‘Extremely uncomfortable’ to 
‘Extremely comfortable’.] 


Question 19 

How effective were module objectives? That is, to what extent did the module 
objectives help you to understand what you would learn in the ‘Space Systems - 
Technologies and Applications’ on-line course. [Five-choice scale, from ‘Extremely 
uncomfortable’ to ‘Extremely comfortable’.] 


Question 20 

How effective were the rules of engagement? That is, to what extent did the rules of 
engagement help you to understand what would be involved in taking the ‘Space 
Systems - Technologies and Applications’ on-line course. [Five-choice scale, from 
‘Extremely uncomfortable’ to ‘Extremely comfortable’.] 


Question 21 

How effective was the course syllabus? That is, to what extent did the course syllabus 
help you to understand what you would learn in the ‘Space Systems - Technologies and 
Applications’ on-line course. [Five-choice scale, from ‘Extremely uncomfortable’ to 
‘Extremely comfortable’.] 


Question 22 


In what areas do you still need help with the web-based technological features (forum, 
email, quizzes, links, etc) used to teach this on-line course? 
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Question 23 

So far, what do you like about this on-line course? 



Question 24 

So far, what are your concerns about this on-line course? 



Question 25 


Please respond to the following question: ‘If I were building an on-line course to teach 
space systems-related instruction, I would.. 



Question 26 

Do you have any suggestions, at this point (in particular, about your orientation to taking 
a course on-line)? 
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APPENDIX C. A-14 QUESTIONNAIRE 



Throughout Module 14 you developed and practiced your skills in the process of 
analyzing, articulating, and evaluating DoD Space Control issues. The statements below 
reflect the learning objectives for Module 14. To what extent do you agree that the 
material presented in the Module enabled you to achieve the learning objectives? 


Question 1 

Identifying key elements of Space Control. (Check one of seven points that fits your 
opinion best.) 

c 1. Strongly disagree, 

r 2. Disagree, 

r 3. Slightly disagree, 

r 4. Neutral. 

r 5. Slightly agree, 

r 6. Agree, 

r 7. Strongly agree. 


Question 2 

Synthesizing other's findings in Forums. [Seven-choice scale, fi-om ‘Strongly disagree’ to 
‘Strongly agree’.] 


Question 3 

Interacting with a subject-matter expert whose knowledge you can use to solve space- 
system problems. [Seven-choice scale, firom ‘Strongly disagree’ to ‘Strongly agree’.] 
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Question 4 


Participating in forums (e.g. discussing, identifying/consulting subject matter experts), to 
evaluate space control issues. [Seven-choice scale, from ‘Strongly disagree’ to ‘Strongly 
agree’.] 


Question 5 

Articulating how the concept of space control relates to Joint Vision 2020. [Seven-choice 
scale, from ‘Strongly disagree’ to ‘Strongly agree’.] 


Question 6 

Evaluating the likelihood that various types of space countermeasures might be used 
across the peace-war continuum. [Seven-choice scale, from ‘Strongly disagree’ to 
‘Strongly agree’.] 


Question 7 

Identifying elements of space systems that are vulnerable to disruption, degradation or 
destruction by adversaries. [Seven-choice scale, from ‘Strongly disagree’ to ‘Strongly 
agree’.] 


Question 8 

In Evaluation A2 we asked you several questions about your comfort level after the 
orientation part of the course. Now, when the course is practically over, we would like 
you to reassess some of these issues, i.e. to think how your comfort level really was 
throughout the course. 

How woxild you rate your comfort level at this point with accessing the Internet? (Check 
one of seven possible answers.) 


r 1. Extremely uncomfortable, 
r 2. Uncomfortable. 
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r 

3. 

Somewhat uncomfortable. 

c 

4. 

Neutral. 

r 

5. 

Somewhat comfortable. 

r 

6. 

Comfortable. 

r 

7. Extremely comfortable. 


Question 9 

How would you rate your comfort level at this point widi sending electronic mail? 
[Seven-choice scale, from ‘Extremely uncomfortable’ to ‘Extremely comfortable’.] 


Question 10 

How would you rate your comfort level at this point with posting comments to the 
Forums? [Seven-choice scale, from ‘Strongly disagree’ to ‘Strongly agree’.] 


Question 11 

How would you rate your comfort level at this point with quiz feature being used to teach 
this course? [Seven-choice scale, from ‘Strongly disagree’ to ‘Strongly agree’.] 


Question 12 

How would you rate your comfort level at this point with the links (i.e., additional web 
sites) feature being used to teach this course? [Seven-choice scde, from ‘Strongly 
disagree’ to ‘Strongly agree’.] 


Question 13 

How would you rate your comfort level at this point with word processing? [Seven- 
choice scale, from ‘Strongly disagree’ to ‘Strongly agree’.] 


Question 14 

Please provide any additional thoughts. 
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APPENDIX D. FINAL SURVEY QUESTIONNAIRE 


Number of Questions: 31 


Please answer the following questions as honestly as you can. The information will be 
compiled and evaluated by a thesis student and will serve as NFS's foundation for 
building future on-line courses. Thanks for taking the time to participate. 

CDR Sue Higgins 


Question 1 

As a result of your overall experience in this course, do you feel that you are able to 
articulate how physical influences on space systems impact our ability to use diose 
systems in military operational situations? 

r 1. Strongly disagree, 

r 2. Disagree, 

r 3. Slightly disagree, 

r 4. Neutral. 

c 5. Slightly agree. 

c 6. Agree, 

r 7. Strongly agree. 


Question 2 

As a result of your experiences in diis course, are you able to articulate how political, 
organizational and economic influences impact how we use space systems in military 
operational settings? [Seven-choice scale, from ‘Strongly disagree’ to ‘Strongly agree’.] 


Question 3 

Do you feel that you are able to articulate how commercial space systems will impact 
military operations? [Seven-choice scale, from ‘Strongly disagree’ to ‘Strongly agree’.] 
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Question 4 


Was the final exam useful in helping you to synthesize the concepts learned in the 
course? 

r 

1. Extremely useless. 

r 

2. Useless. 

r 

3. Slightly useless. 

r 

4. Neutral. 

r 

5. Slightly useful. 

r 

6. Useful. 

r 

7. Extremely useful. 

Question 5 


I feel confident in my ability to access appropriate resources, such as subject matter 
experts, documentation of websites, or late breaking news for evaluating uses of space 
systems in operational situations. 

r 

1. Strongly disagree. 

r 

2. Disagree. 

r 

3. Slightly disagree. 

r 

4. Neutral. 

r 

5. Slightly agree. 

r 

6. Agree. 

r 

7. Strongly agree. 

Question 6 


I feel confident in my ability to get involved in open discussions with my peers to weigh 
issues related to using space systems in operational settings. [Seven-choice scale, from 
‘Strongly disagree’ to ‘Strongly agree’.] 


288 





Question 7 


How useful were the guidelines provided in the course homepage to your ability to 
succeed? 


r 1. Extremely useless, 
r 2. Useless, 

r 3. Slightly useless. 

r 4. Neutral. 

r 5. Slightly useful, 

r 6. Useful, 

r 7. Extremely useful. 


Question 8 

How useful were guidelines provided at the course homepage for posting substantive 
responses to the Forums? [Seven-choice scale, from ‘Extremely useless’ to ‘Extremely 
useful’.] 


Question 9 

How useful was the textbook in helping you understand the course concepts, objectives 
and principles? [Seven-choice scale, from ‘Extremely useless’ to ‘Extremely useful’.] 


Question 10 

How useful were the on-line Module readings in helping you to understand the course 
concepts, objectives and principles? [Seven-choice scale, from ‘Extremely useless’ to 
‘Extremely useful’.] 


Question 11 

How useful were the linked websites in helping you to understand the course concepts, 
objectives and principles? [Seven-choice scale, from ‘Extremely useless’ to ‘Extremely 
useful’.] 
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Question 12 

How useful were the Forum exercises in helping you to xmderstand the subject matter? 
[Seven-choice scale, from ‘Extremely useless’ to ‘Extremely useful’.] 


Question 13 

How useful were the Quizzes (2, 3/4/5, 6/7) for providing feedback in your imderstanding 
of the material for those Modules? (Try to separate the technical problems with the early 
quiz from your answer.) [Seven-choice scale, from ‘Extremely useless’ to ‘Extremely 
useful’.] 


Question 14 

How did you like the Forum assignments? 

c 1. Strongly disliked, 

r 2. Disliked, 

r 3. Slightly disliked, 

r 4. Neutral, 

r 5. Slightly liked, 

r 6. Liked, 

r 7. Strongly liked. 


Question 15 

How did you like the textbook reading assignments? [Seven-choice scale, from ‘Strongly 
disliked’ to ‘Strongly liked’.] 


Question 16 

How did you like the Module readings? [Seven-choice scale, from ‘Strongly disliked’ to 
‘Strongly liked’.] 
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Question 17 


How did you like the website links that were provided? [Seven-choice scale, from 
‘Strongly disliked’ to ‘Strongly liked’.] 


Question 18 

What other instructional materials, if any, should be included in the course? 



Question 19 

Describe your reaction to the instructional strategies used to teach the subject matter 
(textbook readings, forums, websites). 



Question 20 

I am certain that I am able to synthesize information from a variety of sources to enable 
me to use space systems in operational situations. 
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r 1. Strongly disagree, 

r 2. Disagree, 

r 3. Slightly disagree. 

r 4. Neutral, 

r 5. Slightly agree, 

r 6. Agree, 

r 7. Strongly agree. 


Question 21 

As a result of this on-line course experience, do you feel that you have changed your 
fundamental approach to learning? Please respond ‘yes’, ‘somewhat’ or ‘no’, and explain 
your answer. 



Question 22 

Did you view the flexibility of this on-line course as beneficial to your ability to succeed 
in this class? 


r 1. Strongly disagree, 

r 2. Disagree. 
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r 3. subtly disagree, 
r 4. Neutral. 

r 5. Slightly agree. 

r 6. Agree. 

r 7. Strongly agree. 


Question 23 

Do you feel that the flexibility of this class helped you succeed in other classes that you 
were taking in the same quarter? [Seven-choice scale, from ‘Strongly disagree’ to 
‘Strongly agree’.] 


Question 24 

Compare the amount of interaction you had with the instructor for this on-line course as 
compared with a similar resident ‘face-to-face’ course (note - this question is about 
‘amoimt’; another question will cover ‘quality of interaction’). Try to consider only the 
amoimt of interaction that occurred on-line (including e-mails)- as opposed to what 
occurred ifrwhen you stopped by the instructor's office. 

In this on-line class I had significanify less interaction with die instructor than I 
would have in a similar face-to-face class. 

In this on-line class I had less interaction with the instructor than I would have in 
a similar face-to-face class. 

In this on-line class I had slightly less interaction with the instructor than I would 
have in a similar face-to-face class. 

The amount of interaction between the instructor and me was about the same. 

In this on-line class I had slightly more interaction with the instructor than I 
would have in a s imil ar face-to-face class. 

In this on-line class I had more interaction with the instructor than I would have 
in a similar face-to-face class. 

In this on-line class I had significantly more interaction with the instructor than I 
would have in a s imil ar face-to-face class. 


1 . 

2 . 

3. 

4. 

5. 

6 . 
7. 


Question 25 

Compare the quality of interaction between yourself and the instructor in this on-line 
class against what you would have expected in a s imil ar "face-to-face" class. 
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P j In this on-line course the quality of interaction between the instructor and myself 
' was of significantly less quality than it would have been in a face-to-face class, 

P 2 ^ on-line course the quality of interaction between the instructor and myself 

■ was of less quality than it would have been in a face-to-face class. 

3 ^ on-line course the quality of interaction between the instructor and myself 

■ was of slightly less qiiality than it would have been in a face-to-face class. 

c 4. The quality of interaction between the instructor and myself was about the same. 

P ^ In this on-line course the quality of interaction between the instructor and myself 

■ was of slightly greater quality than it would have been in a face-to-face class. 

^ g In this on-line course the quality of interaction between the instructor and myself 
‘ was of greater quality than it would have been in a face-to-face class. 

In this on-line course the quality of interaction between the instructor and myself 
^ 7. was of significantly greater quality than it would have been in a face-to-face 

class. 


Question 26 

Compare the amount of interaction you had with other students for this on-line course as 
compared with a similar resident ‘face-to-face’ course (note - this question is about 
‘amount’; another question will cover ‘quality of interaction’.) 


P j In this on-line course I had significantly less interaction with other students than 

■ in a similar face-to-face comrse. 

2 ^ on-line course I had less interaction with other students than in a similar 

■ face-to-face course. 

P 2 In this on-line course I had /ess interaction with other students than in a 

similar face-to-face covurse. 

c 4 . The amount of interaction was //te sflwe. 

^ ^ In this on-line course I had slightly more interaction with other students than in a 

similar face-to-face course. 

^ g In this on-line course I had more interaction with other students than in a similar 

■ face-to-face course. 

^ 7 ^ on-line course I had significantly more interaction with other students 

■ than in a similar face-to face course. 


Question 27 
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Compare the quality of interaction you had with other students for this on-line course as 
compared with a similar resident ‘face-to-face’ course. 


^ j In this on-line course the quality of interaction between myself and other students 
■ was of significantly less quality than it would have been in a face-to-face class. 

P 2 In this on-line course the quality of interaction between myself and other students 
' was of less quality than it would have been m a face-to-face class. 

P 2 ^ on-line course the quality of interaction between myself and other students 

' was of slightly less quality than it would have been in a face-to-face class. 

r 4. The quality of interaction was aAoiiriftejame. 


r 

r 


r 


^ In this on-line course the quality of interaction between myself and other students 
' was of slightly greater quality than it would have been in a face-to-face class. 

^ In this on-line course the quality of interaction between myself and other students 
’ was of greater quality than it would have been in a face-to-face class. 

In this on-line course the quality of interaction between myself and other students 
7. was of significantly greater quality than it would have been m a face-to-face 
class. 


Question 28 

Describe your reaction to the media used to present the instruction (i.e. online delivery 
instructional technologies.) 



Question 29 

If you wish to add any other comments about the course, use this space. 
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Question 30 (Copyright 1998 © William A. Long and Charles D. Dziuban) 

This question will try to capture your personal traits that are related to your on-line 
learning style. These questions are very easy to answer, but may take some effort to think 
honestly about yourself. We stress that your responses will be used anonymously for 
research purposes only. The following sets represent main types of on-line learning 
styles. The types are not what people think they should be like, or what sounds like a 
desirable description, but rather as people really are. Choose what was closest to you as 
you went through this on-line course. 

Highly energized and action-oriented. Little need for approval; unconcerned with 
^ 1. who they please. Put thinking into immediate action. Very frank, speaks out 

freely. Has no problems confronting people. 

Lower energy level. Little need for approval - unconcerned with pleasing others. 
^ '2. Independent & strong-willed. Prefers to work alone. May resist pressure from 
authority. Independent thinker. 

Highly energized and productive. Strongly motivated by approval. Sensitive to 
the wishes of others. Translates energies into constructive tasks. Deeply values 
^ 3. close bonds with others. Some difficulty dealing with direct confrontation. 

Highly idealistic, setting lofty goals for themselves. Foster harmonious 
relationships. 

Lower energy level. Needs approval - concerned with pleasing others. Rarely 
^ 4. shows anger or resentment. Very sensitive to the feelings of others. Very 

compliant and loyal. Forms strong attachments. Gives and thrives on affection. 


Question 31 (Copyright 1998 ©William A. Long and Charles D. Dziuban) 

Please read carefully the descriptions in the four boxes below. Although not all behaviors 
may fit you exactly, select as many boxes that you feel apply to you. Here you may pick 
from none to four boxes. 
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n 

n 


j Thinks of all possibilities and contingencies before venturing into activities. 
‘What if person. May see the negative side of things. Unwilling to take risks. 

Highly organized and methodical. Strongly motivated to jfinish tasks. 
2. Perfectionist. Tends to form habits. Extremely diligent in work habits. May be 
mildly ritualistic. 

2 Sometimes ejqilosive and quick tempered. Sharp tongued. Very frank. May act 
■ without thinking. 

^ Dramatic. May have wide mood swings. May overreact in some situations. Can 
‘ have emotional outbursts. Artistically inclined. Devalues work routine. 
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APPENDIX E. INTERVIEWS 


INTRODUCTION 

SS3011 on-line course has finished just a couple of months 
ago. Now you have nice time distance from the course, so your 
impressions and opinion about it are more clear and final. The 
purpose of this brief interview is to try to capture some of your 
deeper insights and conclusions about your e^erience of that 
course. 

As in all four on-line evaluation questionnaires you 
answered during the course, your answers will be used only for my 
thesis research and your identity will be protected. Please answer 
them straightforward and comment as you wish. 


QUESTIONS 

1. In general, could you estimate have you learned more from the on-line designed 
course than you would learn from a face-to-face SS3011 course? I’m asking you 
about successful learning of subject matter taught through distributed learning, 
not about particular instructor or anything else. 

2. Students’ answers about forums exercises were very different, in terms of forum 
exercises being useful in helping to understand the subject matter. What is your 
final opinion about their real usefulness in facilitating your learning? Also, 
comment how much you liked using forums. 

3. The same question about Quizzes - please comment how useful they were for 
you and how much you liked them! 

4. Students’ answers to question about how they saw the amount of interaction 
with the instructor during the SS3011 course, compared to a similar resident 
(face-to-face) course, were very different. Please comment your view. 
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5. I want to ask you similar question about the quality of students* interaction 
with the instructor, compared to a resident course. How you saw it? 

6. One of most controversial questions, according to students’ answers, was about 
the amount of interaction between the students, compared to a face-to-face 
course. Some thought SS3011 offered significantly less interaction, some others - 
significantly more. What’s your comment? 

7. The last question about interactions covers the issue of quality of interaction 
between the students, what is important for the learning process. How would you 
comment it, in sense of comparing the quality of interaction between students in 
on-line SS3011 with interaction between students in some similar face-to-face 
course? 

8. If you have an opportunity to choose between on-line and face-to-face section of 
some other class in yom curriculum, what would you do? (Examples: I would 
surely go online; depends on nature of the course; depends on my schedule; 
surely stay in face-to-face section...) 

9. Please tell us any advice/recommendation/suggestion you have for CDR 
Higgins or any other NPS or Navy distributed learning instructor. (This question 
is very open-ended!) 

10. This course was not only a space-systems course, but also a new experience for 
most of the students in the section. Looking at the course from this time distance 
and in general, do you think you’ve gained some other benefits besides the 
course objectives themselves? Please comment. 

11.1 asked you really many questions in the four online questionnaires and in this 
interview. If you think there’s something else I shoiild know, but you haven’t 
comment it yet, please feel firee to do it now. 

At the end, I just want to express my appreciation for your time, patience and 
efforts. I will do my best to make it maximally beneficial for future students, instructors 
and NPS in general, what your answers made possible. Thank you! 
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